2024-03-25T16:21:25.408 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-25T16:21:25.409 DEBUG:teuthology.run:Teuthology command: teuthology --name amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi --archive /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621273 --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} --verbose --owner scheduled_amathuri@teuthology -- /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621273/orig.config.yaml 2024-03-25T16:21:25.453 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T16:21:25.531 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621273 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7621273' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr2: false ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 6785 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi155.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG9GnDhi3qlVC9dImWU7K2yGAWDRaZmta/anqc1QhHI2yQmtpqA6J5ARO+7/Ii0+YsV6tYMdaqBJUklrT1zU160= smithi167.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGkR3F73DIi0cpF43Ju4TRu8aT0j6LuPuo7GMORA5vpzJkhzAQzfFd85n98iFpyhxZ9s35yGY/5pPQTjWg7QfRc= tasks: - install: null - ceph: null - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-25_13:28:33 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-25T16:21:25.531 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-25T16:21:25.532 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-25T16:21:25.532 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-25T16:21:25.534 INFO:teuthology.task.internal:Checking packages... 2024-03-25T16:21:25.558 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-25T16:21:25.558 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-25T16:21:25.558 INFO:teuthology.packaging:ref: None 2024-03-25T16:21:25.558 INFO:teuthology.packaging:tag: None 2024-03-25T16:21:25.558 INFO:teuthology.packaging:branch: main 2024-03-25T16:21:25.558 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:21:25.558 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-25T16:21:26.029 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2381.g44a9b520 2024-03-25T16:21:26.031 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-25T16:21:26.038 INFO:teuthology.task.internal:no buildpackages task found 2024-03-25T16:21:26.038 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-25T16:21:26.045 INFO:teuthology.task.internal:Saving configuration 2024-03-25T16:21:26.062 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-25T16:21:26.068 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T16:21:26.093 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi155.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621273', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:16:04.241597', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG9GnDhi3qlVC9dImWU7K2yGAWDRaZmta/anqc1QhHI2yQmtpqA6J5ARO+7/Ii0+YsV6tYMdaqBJUklrT1zU160='} 2024-03-25T16:21:26.114 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi167.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621273', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:16:04.243555', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGkR3F73DIi0cpF43Ju4TRu8aT0j6LuPuo7GMORA5vpzJkhzAQzfFd85n98iFpyhxZ9s35yGY/5pPQTjWg7QfRc='} 2024-03-25T16:21:26.115 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-25T16:21:26.198 INFO:teuthology.task.internal:roles: ubuntu@smithi155.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-25T16:21:26.198 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-25T16:21:26.198 INFO:teuthology.run_tasks:Running task console_log... 2024-03-25T16:21:26.303 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f138dea90d0>, signals=[15]) 2024-03-25T16:21:26.303 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-25T16:21:26.309 INFO:teuthology.task.internal:Opening connections... 2024-03-25T16:21:26.309 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi155.front.sepia.ceph.com 2024-03-25T16:21:26.311 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:21:26.413 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi167.front.sepia.ceph.com 2024-03-25T16:21:26.413 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:21:26.507 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-25T16:21:26.514 DEBUG:teuthology.orchestra.run.smithi155:> uname -m 2024-03-25T16:21:26.532 INFO:teuthology.orchestra.run.smithi155.stdout:x86_64 2024-03-25T16:21:26.532 DEBUG:teuthology.orchestra.run.smithi155:> cat /etc/os-release 2024-03-25T16:21:26.592 INFO:teuthology.orchestra.run.smithi155.stdout:NAME="CentOS Stream" 2024-03-25T16:21:26.593 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION="9" 2024-03-25T16:21:26.593 INFO:teuthology.orchestra.run.smithi155.stdout:ID="centos" 2024-03-25T16:21:26.593 INFO:teuthology.orchestra.run.smithi155.stdout:ID_LIKE="rhel fedora" 2024-03-25T16:21:26.593 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION_ID="9" 2024-03-25T16:21:26.593 INFO:teuthology.orchestra.run.smithi155.stdout:PLATFORM_ID="platform:el9" 2024-03-25T16:21:26.593 INFO:teuthology.orchestra.run.smithi155.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T16:21:26.593 INFO:teuthology.orchestra.run.smithi155.stdout:ANSI_COLOR="0;31" 2024-03-25T16:21:26.593 INFO:teuthology.orchestra.run.smithi155.stdout:LOGO="fedora-logo-icon" 2024-03-25T16:21:26.593 INFO:teuthology.orchestra.run.smithi155.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T16:21:26.593 INFO:teuthology.orchestra.run.smithi155.stdout:HOME_URL="https://centos.org/" 2024-03-25T16:21:26.593 INFO:teuthology.orchestra.run.smithi155.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T16:21:26.594 INFO:teuthology.orchestra.run.smithi155.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T16:21:26.594 INFO:teuthology.orchestra.run.smithi155.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T16:21:26.594 INFO:teuthology.lock.ops:Updating smithi155.front.sepia.ceph.com on lock server 2024-03-25T16:21:26.621 DEBUG:teuthology.orchestra.run.smithi167:> uname -m 2024-03-25T16:21:26.638 INFO:teuthology.orchestra.run.smithi167.stdout:x86_64 2024-03-25T16:21:26.639 DEBUG:teuthology.orchestra.run.smithi167:> cat /etc/os-release 2024-03-25T16:21:26.699 INFO:teuthology.orchestra.run.smithi167.stdout:NAME="CentOS Stream" 2024-03-25T16:21:26.699 INFO:teuthology.orchestra.run.smithi167.stdout:VERSION="9" 2024-03-25T16:21:26.699 INFO:teuthology.orchestra.run.smithi167.stdout:ID="centos" 2024-03-25T16:21:26.699 INFO:teuthology.orchestra.run.smithi167.stdout:ID_LIKE="rhel fedora" 2024-03-25T16:21:26.699 INFO:teuthology.orchestra.run.smithi167.stdout:VERSION_ID="9" 2024-03-25T16:21:26.699 INFO:teuthology.orchestra.run.smithi167.stdout:PLATFORM_ID="platform:el9" 2024-03-25T16:21:26.700 INFO:teuthology.orchestra.run.smithi167.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T16:21:26.700 INFO:teuthology.orchestra.run.smithi167.stdout:ANSI_COLOR="0;31" 2024-03-25T16:21:26.700 INFO:teuthology.orchestra.run.smithi167.stdout:LOGO="fedora-logo-icon" 2024-03-25T16:21:26.700 INFO:teuthology.orchestra.run.smithi167.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T16:21:26.700 INFO:teuthology.orchestra.run.smithi167.stdout:HOME_URL="https://centos.org/" 2024-03-25T16:21:26.700 INFO:teuthology.orchestra.run.smithi167.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T16:21:26.700 INFO:teuthology.orchestra.run.smithi167.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T16:21:26.700 INFO:teuthology.orchestra.run.smithi167.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T16:21:26.700 INFO:teuthology.lock.ops:Updating smithi167.front.sepia.ceph.com on lock server 2024-03-25T16:21:26.721 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-25T16:21:26.729 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-25T16:21:26.734 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-25T16:21:26.734 DEBUG:teuthology.orchestra.run.smithi155:> test '!' -e /home/ubuntu/cephtest 2024-03-25T16:21:26.736 DEBUG:teuthology.orchestra.run.smithi167:> test '!' -e /home/ubuntu/cephtest 2024-03-25T16:21:26.759 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-25T16:21:26.765 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-25T16:21:26.765 DEBUG:teuthology.orchestra.run.smithi155:> test -z $(ls -A /var/lib/ceph) 2024-03-25T16:21:26.797 DEBUG:teuthology.orchestra.run.smithi167:> test -z $(ls -A /var/lib/ceph) 2024-03-25T16:21:26.907 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-25T16:21:26.961 INFO:teuthology.run_tasks:Running task kernel... 2024-03-25T16:21:26.976 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-25T16:21:26.976 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-25T16:21:26.976 DEBUG:teuthology.orchestra.run.smithi155:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T16:21:26.977 DEBUG:teuthology.orchestra.run.smithi167:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T16:21:26.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:21:26.993 DEBUG:teuthology.orchestra.run.smithi155:> uname -r 2024-03-25T16:21:26.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:21:26.995 DEBUG:teuthology.orchestra.run.smithi167:> uname -r 2024-03-25T16:21:27.054 INFO:teuthology.orchestra.run.smithi155.stdout:5.14.0-361.el9.x86_64 2024-03-25T16:21:27.054 INFO:teuthology.task.kernel:Running kernel on smithi155: 5.14.0-361.el9.x86_64 2024-03-25T16:21:27.055 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum install -y kernel 2024-03-25T16:21:27.056 INFO:teuthology.orchestra.run.smithi167.stdout:5.14.0-361.el9.x86_64 2024-03-25T16:21:27.056 INFO:teuthology.task.kernel:Running kernel on smithi167: 5.14.0-361.el9.x86_64 2024-03-25T16:21:27.056 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum install -y kernel 2024-03-25T16:21:29.245 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-25T16:21:29.268 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-25T16:21:30.854 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS-9 - BaseOS 4.0 MB/s | 2.0 MB 00:00 2024-03-25T16:21:30.935 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS-9 - BaseOS 3.5 MB/s | 2.0 MB 00:00 2024-03-25T16:21:34.834 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS Stream 9 - BaseOS 2.3 MB/s | 8.0 MB 00:03 2024-03-25T16:21:35.461 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - BaseOS 1.9 MB/s | 8.0 MB 00:04 2024-03-25T16:21:37.803 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - AppStream 14 MB/s | 19 MB 00:01 2024-03-25T16:21:42.090 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - CRB 7.4 MB/s | 6.1 MB 00:00 2024-03-25T16:21:43.646 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 16 kB 00:00 2024-03-25T16:21:45.641 INFO:teuthology.orchestra.run.smithi155.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-25T16:21:50.505 INFO:teuthology.orchestra.run.smithi155.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-03-25T16:21:50.825 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS Stream 9 - AppStream 1.3 MB/s | 19 MB 00:14 2024-03-25T16:21:52.620 INFO:teuthology.orchestra.run.smithi155.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T16:21:52.621 INFO:teuthology.orchestra.run.smithi155.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T16:21:52.713 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-25T16:21:52.717 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T16:21:52.717 INFO:teuthology.orchestra.run.smithi155.stdout: Package Architecture Version Repository Size 2024-03-25T16:21:52.717 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T16:21:52.717 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-25T16:21:52.717 INFO:teuthology.orchestra.run.smithi155.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-25T16:21:52.717 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-03-25T16:21:52.717 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-25T16:21:52.717 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-25T16:21:52.718 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-25T16:21:52.718 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:21:52.718 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-25T16:21:52.718 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T16:21:52.718 INFO:teuthology.orchestra.run.smithi155.stdout:Install 4 Packages 2024-03-25T16:21:52.718 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:21:52.718 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 76 M 2024-03-25T16:21:52.719 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 123 M 2024-03-25T16:21:52.719 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-25T16:21:52.898 INFO:teuthology.orchestra.run.smithi155.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 373 kB/s | 44 kB 00:00 2024-03-25T16:21:53.482 INFO:teuthology.orchestra.run.smithi155.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 21 MB/s | 15 MB 00:00 2024-03-25T16:21:54.274 INFO:teuthology.orchestra.run.smithi155.stdout:(3/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 22 MB/s | 33 MB 00:01 2024-03-25T16:21:54.524 INFO:teuthology.orchestra.run.smithi155.stdout:(4/4): kernel-modules-core-5.14.0-430.el9.x86_6 17 MB/s | 27 MB 00:01 2024-03-25T16:21:54.524 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:21:54.525 INFO:teuthology.orchestra.run.smithi155.stdout:Total 42 MB/s | 76 MB 00:01 2024-03-25T16:21:54.768 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.1 MB 00:00 2024-03-25T16:21:55.187 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-25T16:21:55.564 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-25T16:21:55.564 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-25T16:21:56.368 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 16 kB 00:00 2024-03-25T16:21:56.577 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-25T16:21:56.578 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-25T16:21:57.286 INFO:teuthology.orchestra.run.smithi167.stdout:Extra Packages for Enterprise Linux 26 MB/s | 21 MB 00:00 2024-03-25T16:21:59.054 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-25T16:21:59.794 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:21:59.881 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:22:02.195 INFO:teuthology.orchestra.run.smithi167.stdout:lab-extras 16 kB/s | 1.7 kB 00:00 2024-03-25T16:22:02.650 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:22:02.790 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:22:04.260 INFO:teuthology.orchestra.run.smithi167.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T16:22:04.260 INFO:teuthology.orchestra.run.smithi167.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T16:22:04.349 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-03-25T16:22:04.355 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T16:22:04.355 INFO:teuthology.orchestra.run.smithi167.stdout: Package Architecture Version Repository Size 2024-03-25T16:22:04.355 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T16:22:04.355 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-03-25T16:22:04.355 INFO:teuthology.orchestra.run.smithi167.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-25T16:22:04.355 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2024-03-25T16:22:04.356 INFO:teuthology.orchestra.run.smithi167.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-25T16:22:04.356 INFO:teuthology.orchestra.run.smithi167.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-25T16:22:04.356 INFO:teuthology.orchestra.run.smithi167.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-25T16:22:04.356 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:22:04.356 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-03-25T16:22:04.356 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T16:22:04.356 INFO:teuthology.orchestra.run.smithi167.stdout:Install 4 Packages 2024-03-25T16:22:04.356 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:22:04.357 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 76 M 2024-03-25T16:22:04.357 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 123 M 2024-03-25T16:22:04.357 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-03-25T16:22:04.563 INFO:teuthology.orchestra.run.smithi167.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 309 kB/s | 44 kB 00:00 2024-03-25T16:22:07.500 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:22:07.542 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:22:07.863 INFO:teuthology.orchestra.run.smithi167.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 4.3 MB/s | 15 MB 00:03 2024-03-25T16:22:09.230 INFO:teuthology.orchestra.run.smithi167.stdout:(3/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 7.0 MB/s | 33 MB 00:04 2024-03-25T16:22:09.863 INFO:teuthology.orchestra.run.smithi167.stdout:(4/4): kernel-modules-core-5.14.0-430.el9.x86_6 5.2 MB/s | 27 MB 00:05 2024-03-25T16:22:09.864 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:22:09.864 INFO:teuthology.orchestra.run.smithi167.stdout:Total 14 MB/s | 76 MB 00:05 2024-03-25T16:22:10.531 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-03-25T16:22:10.921 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-03-25T16:22:10.922 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-03-25T16:22:11.927 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-03-25T16:22:11.927 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-03-25T16:22:12.305 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:22:14.401 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-03-25T16:22:15.012 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:22:15.089 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:22:17.826 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:22:17.968 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:22:22.710 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:22:22.826 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:22:27.601 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:22:41.850 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:22:41.851 INFO:teuthology.orchestra.run.smithi155.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T16:22:41.851 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:22:41.899 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:22:42.265 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:22:42.265 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:22:42.265 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:22:42.265 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:22:42.682 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:22:42.682 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:22:42.682 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-25T16:22:42.683 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-25T16:22:42.683 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-25T16:22:42.683 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-25T16:22:42.683 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-25T16:22:42.683 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:22:42.683 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-25T16:22:43.001 DEBUG:teuthology.orchestra.run.smithi155:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:22:43.094 INFO:teuthology.orchestra.run.smithi155.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:22:43.094 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:22:43.094 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-25T16:22:43.094 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T16:22:43.094 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T16:22:43.094 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-25T16:22:43.095 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi155.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T16:22:43.095 DEBUG:teuthology.orchestra.run.smithi155:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:22:43.175 INFO:teuthology.orchestra.run.smithi155.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:22:43.175 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:22:43.175 DEBUG:teuthology.orchestra.run.smithi155:> sudo rpm -qi grub2-tools 2024-03-25T16:22:43.216 INFO:teuthology.orchestra.run.smithi155.stdout:Name : grub2-tools 2024-03-25T16:22:43.216 INFO:teuthology.orchestra.run.smithi155.stdout:Epoch : 1 2024-03-25T16:22:43.216 INFO:teuthology.orchestra.run.smithi155.stdout:Version : 2.06 2024-03-25T16:22:43.216 INFO:teuthology.orchestra.run.smithi155.stdout:Release : 68.el9 2024-03-25T16:22:43.216 INFO:teuthology.orchestra.run.smithi155.stdout:Architecture: x86_64 2024-03-25T16:22:43.216 INFO:teuthology.orchestra.run.smithi155.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T16:22:43.216 INFO:teuthology.orchestra.run.smithi155.stdout:Group : Unspecified 2024-03-25T16:22:43.216 INFO:teuthology.orchestra.run.smithi155.stdout:Size : 8264254 2024-03-25T16:22:43.216 INFO:teuthology.orchestra.run.smithi155.stdout:License : GPLv3+ 2024-03-25T16:22:43.216 INFO:teuthology.orchestra.run.smithi155.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T16:22:43.216 INFO:teuthology.orchestra.run.smithi155.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T16:22:43.217 INFO:teuthology.orchestra.run.smithi155.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T16:22:43.217 INFO:teuthology.orchestra.run.smithi155.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T16:22:43.217 INFO:teuthology.orchestra.run.smithi155.stdout:Packager : builder@centos.org 2024-03-25T16:22:43.217 INFO:teuthology.orchestra.run.smithi155.stdout:Vendor : CentOS 2024-03-25T16:22:43.217 INFO:teuthology.orchestra.run.smithi155.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T16:22:43.217 INFO:teuthology.orchestra.run.smithi155.stdout:Summary : Support tools for GRUB. 2024-03-25T16:22:43.217 INFO:teuthology.orchestra.run.smithi155.stdout:Description : 2024-03-25T16:22:43.217 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:22:43.217 INFO:teuthology.orchestra.run.smithi155.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T16:22:43.217 INFO:teuthology.orchestra.run.smithi155.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T16:22:43.217 INFO:teuthology.orchestra.run.smithi155.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T16:22:43.217 INFO:teuthology.orchestra.run.smithi155.stdout:hardware devices. 2024-03-25T16:22:43.217 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:22:43.217 INFO:teuthology.orchestra.run.smithi155.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T16:22:43.218 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T16:22:43.218 INFO:teuthology.task.kernel:Updating grub on smithi155 to boot 5.14.0-430.el9.x86_64 2024-03-25T16:22:43.219 DEBUG:teuthology.orchestra.run.smithi155:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T16:22:43.879 INFO:teuthology.orchestra.run.smithi155.stderr:Generating grub configuration file ... 2024-03-25T16:22:45.273 INFO:teuthology.orchestra.run.smithi155.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T16:22:45.328 INFO:teuthology.orchestra.run.smithi155.stderr:done 2024-03-25T16:22:45.331 DEBUG:teuthology.orchestra.run.smithi155:> mktemp 2024-03-25T16:22:45.346 INFO:teuthology.orchestra.run.smithi155.stdout:/tmp/tmp.L06c63qfvU 2024-03-25T16:22:45.347 DEBUG:teuthology.orchestra.run.smithi155:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.L06c63qfvU 2024-03-25T16:22:45.415 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 0666 /tmp/tmp.L06c63qfvU 2024-03-25T16:22:45.556 DEBUG:teuthology.orchestra.remote:smithi155:/tmp/tmp.L06c63qfvU is 6KB 2024-03-25T16:22:45.604 DEBUG:teuthology.orchestra.run.smithi155:> rm -fr /tmp/tmp.L06c63qfvU 2024-03-25T16:22:45.618 DEBUG:teuthology.orchestra.run.smithi155:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T16:22:45.684 INFO:teuthology.orchestra.run.smithi155.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T16:22:45.684 INFO:teuthology.orchestra.run.smithi155.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T16:22:45.684 INFO:teuthology.orchestra.run.smithi155.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T16:22:45.684 INFO:teuthology.orchestra.run.smithi155.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-25T16:22:45.685 DEBUG:teuthology.orchestra.run.smithi155:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-25T16:22:45.837 DEBUG:teuthology.orchestra.run.smithi155:> sudo shutdown -r now 2024-03-25T16:22:57.008 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:22:57.008 INFO:teuthology.orchestra.run.smithi167.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T16:22:57.009 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:22:57.052 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:22:57.441 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:22:57.441 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:22:57.441 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:22:57.441 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:22:57.861 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:22:57.861 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:22:57.861 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-03-25T16:22:57.861 INFO:teuthology.orchestra.run.smithi167.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-25T16:22:57.861 INFO:teuthology.orchestra.run.smithi167.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-25T16:22:57.861 INFO:teuthology.orchestra.run.smithi167.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-25T16:22:57.861 INFO:teuthology.orchestra.run.smithi167.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-25T16:22:57.861 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:22:57.862 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-03-25T16:22:58.179 DEBUG:teuthology.orchestra.run.smithi167:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:22:58.278 INFO:teuthology.orchestra.run.smithi167.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:22:58.278 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:22:58.279 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-25T16:22:58.279 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T16:22:58.279 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T16:22:58.279 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-25T16:22:58.279 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi167.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T16:22:58.279 DEBUG:teuthology.orchestra.run.smithi167:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:22:58.360 INFO:teuthology.orchestra.run.smithi167.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:22:58.360 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:22:58.360 DEBUG:teuthology.orchestra.run.smithi167:> sudo rpm -qi grub2-tools 2024-03-25T16:22:58.394 INFO:teuthology.orchestra.run.smithi167.stdout:Name : grub2-tools 2024-03-25T16:22:58.395 INFO:teuthology.orchestra.run.smithi167.stdout:Epoch : 1 2024-03-25T16:22:58.395 INFO:teuthology.orchestra.run.smithi167.stdout:Version : 2.06 2024-03-25T16:22:58.395 INFO:teuthology.orchestra.run.smithi167.stdout:Release : 68.el9 2024-03-25T16:22:58.395 INFO:teuthology.orchestra.run.smithi167.stdout:Architecture: x86_64 2024-03-25T16:22:58.395 INFO:teuthology.orchestra.run.smithi167.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T16:22:58.395 INFO:teuthology.orchestra.run.smithi167.stdout:Group : Unspecified 2024-03-25T16:22:58.395 INFO:teuthology.orchestra.run.smithi167.stdout:Size : 8264254 2024-03-25T16:22:58.395 INFO:teuthology.orchestra.run.smithi167.stdout:License : GPLv3+ 2024-03-25T16:22:58.395 INFO:teuthology.orchestra.run.smithi167.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T16:22:58.395 INFO:teuthology.orchestra.run.smithi167.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T16:22:58.395 INFO:teuthology.orchestra.run.smithi167.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T16:22:58.395 INFO:teuthology.orchestra.run.smithi167.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T16:22:58.396 INFO:teuthology.orchestra.run.smithi167.stdout:Packager : builder@centos.org 2024-03-25T16:22:58.396 INFO:teuthology.orchestra.run.smithi167.stdout:Vendor : CentOS 2024-03-25T16:22:58.396 INFO:teuthology.orchestra.run.smithi167.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T16:22:58.396 INFO:teuthology.orchestra.run.smithi167.stdout:Summary : Support tools for GRUB. 2024-03-25T16:22:58.396 INFO:teuthology.orchestra.run.smithi167.stdout:Description : 2024-03-25T16:22:58.396 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:22:58.396 INFO:teuthology.orchestra.run.smithi167.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T16:22:58.396 INFO:teuthology.orchestra.run.smithi167.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T16:22:58.396 INFO:teuthology.orchestra.run.smithi167.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T16:22:58.396 INFO:teuthology.orchestra.run.smithi167.stdout:hardware devices. 2024-03-25T16:22:58.396 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:22:58.396 INFO:teuthology.orchestra.run.smithi167.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T16:22:58.397 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T16:22:58.397 INFO:teuthology.task.kernel:Updating grub on smithi167 to boot 5.14.0-430.el9.x86_64 2024-03-25T16:22:58.397 DEBUG:teuthology.orchestra.run.smithi167:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T16:22:59.121 INFO:teuthology.orchestra.run.smithi167.stderr:Generating grub configuration file ... 2024-03-25T16:23:00.507 INFO:teuthology.orchestra.run.smithi167.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T16:23:00.568 INFO:teuthology.orchestra.run.smithi167.stderr:done 2024-03-25T16:23:00.573 DEBUG:teuthology.orchestra.run.smithi167:> mktemp 2024-03-25T16:23:00.588 INFO:teuthology.orchestra.run.smithi167.stdout:/tmp/tmp.6STaS1JBQA 2024-03-25T16:23:00.588 DEBUG:teuthology.orchestra.run.smithi167:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.6STaS1JBQA 2024-03-25T16:23:00.656 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod 0666 /tmp/tmp.6STaS1JBQA 2024-03-25T16:23:00.800 DEBUG:teuthology.orchestra.remote:smithi167:/tmp/tmp.6STaS1JBQA is 6KB 2024-03-25T16:23:00.849 DEBUG:teuthology.orchestra.run.smithi167:> rm -fr /tmp/tmp.6STaS1JBQA 2024-03-25T16:23:00.863 DEBUG:teuthology.orchestra.run.smithi167:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T16:23:00.928 INFO:teuthology.orchestra.run.smithi167.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T16:23:00.928 INFO:teuthology.orchestra.run.smithi167.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T16:23:00.928 INFO:teuthology.orchestra.run.smithi167.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T16:23:00.928 INFO:teuthology.orchestra.run.smithi167.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-25T16:23:00.930 DEBUG:teuthology.orchestra.run.smithi167:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-25T16:23:01.067 DEBUG:teuthology.orchestra.run.smithi167:> sudo shutdown -r now 2024-03-25T16:23:15.854 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-25T16:23:15.855 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-25T16:23:15.856 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:23:31.085 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-25T16:23:31.086 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2024-03-25T16:23:31.086 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:23:34.174 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-25T16:23:43.182 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-25T16:23:43.184 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:23:46.238 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-25T16:23:49.535 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.167 2024-03-25T16:23:58.246 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-25T16:23:58.247 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:23:58.536 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2024-03-25T16:23:58.537 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:24:01.598 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.167 2024-03-25T16:24:13.610 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2024-03-25T16:24:13.611 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:24:16.670 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-25T16:24:31.683 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-25T16:24:31.684 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:24:34.754 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-25T16:24:52.770 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-25T16:24:52.771 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:24:53.262 DEBUG:teuthology.orchestra.run.smithi155:> true 2024-03-25T16:24:53.657 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-25T16:24:53.657 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-430.el9.x86_64"... 2024-03-25T16:24:53.658 DEBUG:teuthology.orchestra.run.smithi155:> uname -r 2024-03-25T16:24:53.673 INFO:teuthology.orchestra.run.smithi155.stdout:5.14.0-430.el9.x86_64 2024-03-25T16:24:53.674 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-25T16:24:53.674 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T16:24:53.675 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T16:24:54.675 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-25T16:24:54.675 DEBUG:teuthology.orchestra.run.smithi155:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T16:24:54.757 INFO:teuthology.orchestra.run.smithi155.stdout:ttyS1 2024-03-25T16:24:54.795 DEBUG:teuthology.parallel:result is None 2024-03-25T16:25:13.613 DEBUG:teuthology.orchestra.remote:timed out 2024-03-25T16:25:28.614 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2024-03-25T16:25:28.615 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:25:28.879 DEBUG:teuthology.orchestra.run.smithi167:> true 2024-03-25T16:25:29.272 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi167.front.sepia.ceph.com' 2024-03-25T16:25:29.273 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-430.el9.x86_64"... 2024-03-25T16:25:29.273 DEBUG:teuthology.orchestra.run.smithi167:> uname -r 2024-03-25T16:25:29.288 INFO:teuthology.orchestra.run.smithi167.stdout:5.14.0-430.el9.x86_64 2024-03-25T16:25:29.289 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-25T16:25:29.289 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T16:25:29.289 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T16:25:30.290 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-25T16:25:30.290 DEBUG:teuthology.orchestra.run.smithi167:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T16:25:30.391 INFO:teuthology.orchestra.run.smithi167.stdout:ttyS1 2024-03-25T16:25:30.429 DEBUG:teuthology.parallel:result is None 2024-03-25T16:25:30.430 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-25T16:25:30.440 INFO:teuthology.task.internal:Creating test directory... 2024-03-25T16:25:30.440 DEBUG:teuthology.orchestra.run.smithi155:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T16:25:30.444 DEBUG:teuthology.orchestra.run.smithi167:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T16:25:30.462 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-25T16:25:30.472 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-25T16:25:30.482 INFO:teuthology.task.internal:Creating archive directory... 2024-03-25T16:25:30.483 DEBUG:teuthology.orchestra.run.smithi155:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T16:25:30.503 DEBUG:teuthology.orchestra.run.smithi167:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T16:25:30.541 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-25T16:25:30.551 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-25T16:25:30.552 DEBUG:teuthology.orchestra.run.smithi155:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T16:25:30.570 DEBUG:teuthology.orchestra.run.smithi167:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T16:25:30.609 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:25:30.618 INFO:teuthology.orchestra.run.smithi167.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:25:30.623 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:25:30.632 INFO:teuthology.orchestra.run.smithi167.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:25:30.634 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-25T16:25:30.642 INFO:teuthology.task.internal:Configuring sudo... 2024-03-25T16:25:30.642 DEBUG:teuthology.orchestra.run.smithi155:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T16:25:30.668 DEBUG:teuthology.orchestra.run.smithi167:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T16:25:30.697 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-25T16:25:30.706 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-25T16:25:30.706 DEBUG:teuthology.orchestra.run.smithi155:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T16:25:30.739 DEBUG:teuthology.orchestra.run.smithi167:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T16:25:30.755 DEBUG:teuthology.orchestra.run.smithi155:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:25:30.831 DEBUG:teuthology.orchestra.run.smithi155:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:25:30.902 DEBUG:teuthology.orchestra.run.smithi155:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:25:30.941 DEBUG:teuthology.orchestra.run.smithi155:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:25:30.987 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-25T16:25:30.987 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T16:25:31.057 DEBUG:teuthology.orchestra.run.smithi167:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:25:31.100 DEBUG:teuthology.orchestra.run.smithi167:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:25:31.154 DEBUG:teuthology.orchestra.run.smithi167:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:25:31.236 DEBUG:teuthology.orchestra.run.smithi167:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:25:31.284 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-03-25T16:25:31.284 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T16:25:31.354 DEBUG:teuthology.orchestra.run.smithi155:> sudo service rsyslog restart 2024-03-25T16:25:31.357 DEBUG:teuthology.orchestra.run.smithi167:> sudo service rsyslog restart 2024-03-25T16:25:31.401 INFO:teuthology.orchestra.run.smithi155.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T16:25:31.439 INFO:teuthology.orchestra.run.smithi167.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T16:25:31.900 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-25T16:25:31.909 INFO:teuthology.task.internal:Starting timer... 2024-03-25T16:25:31.909 INFO:teuthology.run_tasks:Running task pcp... 2024-03-25T16:25:31.933 INFO:teuthology.run_tasks:Running task selinux... 2024-03-25T16:25:31.941 DEBUG:teuthology.orchestra.run.smithi155:> sudo service auditd rotate 2024-03-25T16:25:31.996 INFO:teuthology.orchestra.run.smithi155.stdout:Rotating logs: 2024-03-25T16:25:31.998 DEBUG:teuthology.orchestra.run.smithi167:> sudo service auditd rotate 2024-03-25T16:25:32.051 INFO:teuthology.orchestra.run.smithi167.stdout:Rotating logs: 2024-03-25T16:25:32.052 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-25T16:25:32.053 DEBUG:teuthology.orchestra.run.smithi155:> /usr/sbin/getenforce 2024-03-25T16:25:32.074 INFO:teuthology.orchestra.run.smithi155.stdout:Permissive 2024-03-25T16:25:32.074 DEBUG:teuthology.orchestra.run.smithi167:> /usr/sbin/getenforce 2024-03-25T16:25:32.096 INFO:teuthology.orchestra.run.smithi167.stdout:Permissive 2024-03-25T16:25:32.096 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi155.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi167.front.sepia.ceph.com': 'permissive'} 2024-03-25T16:25:32.096 DEBUG:teuthology.orchestra.run.smithi155:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T16:25:32.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:25:32.143 DEBUG:teuthology.orchestra.run.smithi167:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T16:25:32.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:25:32.172 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-25T16:25:32.173 DEBUG:teuthology.orchestra.run.smithi155:> sudo /usr/sbin/setenforce permissive 2024-03-25T16:25:32.217 DEBUG:teuthology.orchestra.run.smithi167:> sudo /usr/sbin/setenforce permissive 2024-03-25T16:25:32.255 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-25T16:25:32.264 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-25T16:25:32.281 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-25T16:25:32.283 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi155.front.sepia.ceph.com,smithi167.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-25T16:29:48.531 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi155.front.sepia.ceph.com'), Remote(name='ubuntu@smithi167.front.sepia.ceph.com')] 2024-03-25T16:29:48.551 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-25T16:29:48.552 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:29:48.649 DEBUG:teuthology.orchestra.run.smithi155:> true 2024-03-25T16:29:48.725 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-25T16:29:48.726 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2024-03-25T16:29:48.726 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:29:48.814 DEBUG:teuthology.orchestra.run.smithi167:> true 2024-03-25T16:29:48.889 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi167.front.sepia.ceph.com' 2024-03-25T16:29:48.890 INFO:teuthology.run_tasks:Running task clock... 2024-03-25T16:29:48.900 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-25T16:29:48.900 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T16:29:48.901 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T16:29:48.903 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T16:29:48.903 DEBUG:teuthology.orchestra.run.smithi167:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T16:29:48.932 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T16:29:48.949 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T16:29:48.961 INFO:teuthology.orchestra.run.smithi167.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T16:29:48.978 INFO:teuthology.orchestra.run.smithi155.stderr:sudo: ntpd: command not found 2024-03-25T16:29:48.982 INFO:teuthology.orchestra.run.smithi167.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T16:29:48.991 INFO:teuthology.orchestra.run.smithi155.stdout:506 Cannot talk to daemon 2024-03-25T16:29:49.008 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T16:29:49.008 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: ntpd: command not found 2024-03-25T16:29:49.021 INFO:teuthology.orchestra.run.smithi167.stdout:506 Cannot talk to daemon 2024-03-25T16:29:49.024 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T16:29:49.038 INFO:teuthology.orchestra.run.smithi167.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T16:29:49.054 INFO:teuthology.orchestra.run.smithi167.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T16:29:49.075 INFO:teuthology.orchestra.run.smithi155.stderr:bash: line 1: ntpq: command not found 2024-03-25T16:29:49.079 INFO:teuthology.orchestra.run.smithi155.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T16:29:49.079 INFO:teuthology.orchestra.run.smithi155.stdout:=============================================================================== 2024-03-25T16:29:49.079 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:29:49.080 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:29:49.080 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:29:49.080 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:29:49.108 INFO:teuthology.orchestra.run.smithi167.stderr:bash: line 1: ntpq: command not found 2024-03-25T16:29:49.112 INFO:teuthology.orchestra.run.smithi167.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T16:29:49.113 INFO:teuthology.orchestra.run.smithi167.stdout:=============================================================================== 2024-03-25T16:29:49.113 INFO:teuthology.orchestra.run.smithi167.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:29:49.113 INFO:teuthology.orchestra.run.smithi167.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:29:49.113 INFO:teuthology.orchestra.run.smithi167.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:29:49.113 INFO:teuthology.orchestra.run.smithi167.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:29:49.113 INFO:teuthology.run_tasks:Running task install... 2024-03-25T16:29:49.122 DEBUG:teuthology.task.install:project ceph 2024-03-25T16:29:49.122 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-25T16:29:49.122 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-25T16:29:49.123 INFO:teuthology.task.install:Using flavor: default 2024-03-25T16:29:49.129 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-03-25T16:29:49.129 INFO:teuthology.task.install:extra packages: [] 2024-03-25T16:29:49.129 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T16:29:49.130 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:29:49.132 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T16:29:49.133 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:29:49.381 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T16:29:49.381 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T16:29:49.391 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T16:29:49.391 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T16:29:49.797 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T16:29:49.798 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-25T16:29:49.798 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T16:29:49.809 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T16:29:49.809 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-03-25T16:29:49.809 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T16:29:49.834 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T16:29:49.835 DEBUG:teuthology.orchestra.run.smithi155:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T16:29:49.838 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T16:29:49.839 DEBUG:teuthology.orchestra.run.smithi167:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T16:29:49.904 DEBUG:teuthology.orchestra.run.smithi155:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T16:29:49.908 DEBUG:teuthology.orchestra.run.smithi167:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T16:29:49.986 DEBUG:teuthology.orchestra.run.smithi155:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T16:29:49.988 DEBUG:teuthology.orchestra.run.smithi167:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T16:29:50.053 INFO:teuthology.orchestra.run.smithi155.stdout:check_obsoletes = 1 2024-03-25T16:29:50.055 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum clean all 2024-03-25T16:29:50.055 INFO:teuthology.orchestra.run.smithi167.stdout:check_obsoletes = 1 2024-03-25T16:29:50.057 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum clean all 2024-03-25T16:29:50.293 INFO:teuthology.orchestra.run.smithi167.stdout:74 files removed 2024-03-25T16:29:50.302 INFO:teuthology.orchestra.run.smithi155.stdout:74 files removed 2024-03-25T16:29:50.320 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-radosgw 2024-03-25T16:29:50.332 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-radosgw 2024-03-25T16:29:51.742 INFO:teuthology.orchestra.run.smithi167.stdout:ceph packages for x86_64 73 kB/s | 82 kB 00:01 2024-03-25T16:29:51.815 INFO:teuthology.orchestra.run.smithi155.stdout:ceph packages for x86_64 69 kB/s | 82 kB 00:01 2024-03-25T16:29:52.642 INFO:teuthology.orchestra.run.smithi167.stdout:ceph noarch packages 12 kB/s | 10 kB 00:00 2024-03-25T16:29:52.815 INFO:teuthology.orchestra.run.smithi155.stdout:ceph noarch packages 11 kB/s | 10 kB 00:00 2024-03-25T16:29:53.518 INFO:teuthology.orchestra.run.smithi167.stdout:ceph source packages 2.3 kB/s | 1.9 kB 00:00 2024-03-25T16:29:53.557 INFO:teuthology.orchestra.run.smithi155.stdout:ceph source packages 2.7 kB/s | 1.9 kB 00:00 2024-03-25T16:29:54.062 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-25T16:29:54.118 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-25T16:29:55.463 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS-9 - BaseOS 6.9 MB/s | 2.0 MB 00:00 2024-03-25T16:29:55.684 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS-9 - BaseOS 5.4 MB/s | 2.0 MB 00:00 2024-03-25T16:29:56.506 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-03-25T16:29:56.675 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-25T16:30:00.151 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - AppStream 7.7 MB/s | 19 MB 00:02 2024-03-25T16:30:04.436 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.1 MB 00:00 2024-03-25T16:30:05.836 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - Extras packages 62 kB/s | 16 kB 00:00 2024-03-25T16:30:06.963 INFO:teuthology.orchestra.run.smithi155.stdout:Extra Packages for Enterprise Linux 20 MB/s | 21 MB 00:01 2024-03-25T16:30:12.088 INFO:teuthology.orchestra.run.smithi155.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-03-25T16:30:13.419 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 19 MB 00:15 2024-03-25T16:30:14.524 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-25T16:30:14.540 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T16:30:14.540 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-03-25T16:30:14.540 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T16:30:14.540 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-25T16:30:14.540 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T16:30:14.540 INFO:teuthology.orchestra.run.smithi155.stdout:Upgrading: 2024-03-25T16:30:14.540 INFO:teuthology.orchestra.run.smithi155.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T16:30:14.541 INFO:teuthology.orchestra.run.smithi155.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:30:14.541 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-03-25T16:30:14.541 INFO:teuthology.orchestra.run.smithi155.stdout: boost-program-options 2024-03-25T16:30:14.541 INFO:teuthology.orchestra.run.smithi155.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T16:30:14.541 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T16:30:14.541 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T16:30:14.541 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T16:30:14.541 INFO:teuthology.orchestra.run.smithi155.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T16:30:14.541 INFO:teuthology.orchestra.run.smithi155.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T16:30:14.541 INFO:teuthology.orchestra.run.smithi155.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T16:30:14.541 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T16:30:14.541 INFO:teuthology.orchestra.run.smithi155.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T16:30:14.542 INFO:teuthology.orchestra.run.smithi155.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T16:30:14.542 INFO:teuthology.orchestra.run.smithi155.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T16:30:14.542 INFO:teuthology.orchestra.run.smithi155.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T16:30:14.542 INFO:teuthology.orchestra.run.smithi155.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T16:30:14.542 INFO:teuthology.orchestra.run.smithi155.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T16:30:14.542 INFO:teuthology.orchestra.run.smithi155.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T16:30:14.542 INFO:teuthology.orchestra.run.smithi155.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T16:30:14.542 INFO:teuthology.orchestra.run.smithi155.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T16:30:14.542 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-argparse 2024-03-25T16:30:14.542 INFO:teuthology.orchestra.run.smithi155.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T16:30:14.542 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T16:30:14.543 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T16:30:14.543 INFO:teuthology.orchestra.run.smithi155.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T16:30:14.543 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T16:30:14.543 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T16:30:14.543 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T16:30:14.543 INFO:teuthology.orchestra.run.smithi155.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-25T16:30:14.543 INFO:teuthology.orchestra.run.smithi155.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-25T16:30:14.543 INFO:teuthology.orchestra.run.smithi155.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T16:30:14.543 INFO:teuthology.orchestra.run.smithi155.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T16:30:14.543 INFO:teuthology.orchestra.run.smithi155.stdout:Installing weak dependencies: 2024-03-25T16:30:14.543 INFO:teuthology.orchestra.run.smithi155.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-25T16:30:14.543 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:30:14.544 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-25T16:30:14.544 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T16:30:14.544 INFO:teuthology.orchestra.run.smithi155.stdout:Install 30 Packages 2024-03-25T16:30:14.544 INFO:teuthology.orchestra.run.smithi155.stdout:Upgrade 2 Packages 2024-03-25T16:30:14.544 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:30:14.544 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 58 M 2024-03-25T16:30:14.545 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-25T16:30:21.289 INFO:teuthology.orchestra.run.smithi155.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 873 kB/s | 5.3 MB 00:06 2024-03-25T16:30:21.389 INFO:teuthology.orchestra.run.smithi155.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 252 kB/s | 25 kB 00:00 2024-03-25T16:30:23.772 INFO:teuthology.orchestra.run.smithi155.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 311 kB/s | 742 kB 00:02 2024-03-25T16:30:25.364 INFO:teuthology.orchestra.run.smithi155.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 313 kB/s | 498 kB 00:01 2024-03-25T16:30:27.112 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS Stream 9 - CRB 613 kB/s | 6.1 MB 00:10 2024-03-25T16:30:28.746 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 16 kB 00:00 2024-03-25T16:30:30.303 INFO:teuthology.orchestra.run.smithi167.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-03-25T16:30:35.206 INFO:teuthology.orchestra.run.smithi167.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2024-03-25T16:30:37.652 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-03-25T16:30:37.655 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T16:30:37.655 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2024-03-25T16:30:37.655 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T16:30:37.655 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-03-25T16:30:37.655 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T16:30:37.656 INFO:teuthology.orchestra.run.smithi167.stdout:Upgrading: 2024-03-25T16:30:37.656 INFO:teuthology.orchestra.run.smithi167.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T16:30:37.656 INFO:teuthology.orchestra.run.smithi167.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:30:37.656 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2024-03-25T16:30:37.656 INFO:teuthology.orchestra.run.smithi167.stdout: boost-program-options 2024-03-25T16:30:37.656 INFO:teuthology.orchestra.run.smithi167.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T16:30:37.656 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T16:30:37.656 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T16:30:37.656 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T16:30:37.656 INFO:teuthology.orchestra.run.smithi167.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T16:30:37.656 INFO:teuthology.orchestra.run.smithi167.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T16:30:37.656 INFO:teuthology.orchestra.run.smithi167.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T16:30:37.656 INFO:teuthology.orchestra.run.smithi167.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T16:30:37.657 INFO:teuthology.orchestra.run.smithi167.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T16:30:37.657 INFO:teuthology.orchestra.run.smithi167.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T16:30:37.657 INFO:teuthology.orchestra.run.smithi167.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T16:30:37.657 INFO:teuthology.orchestra.run.smithi167.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T16:30:37.657 INFO:teuthology.orchestra.run.smithi167.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T16:30:37.657 INFO:teuthology.orchestra.run.smithi167.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T16:30:37.657 INFO:teuthology.orchestra.run.smithi167.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T16:30:37.657 INFO:teuthology.orchestra.run.smithi167.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T16:30:37.657 INFO:teuthology.orchestra.run.smithi167.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T16:30:37.657 INFO:teuthology.orchestra.run.smithi167.stdout: python3-ceph-argparse 2024-03-25T16:30:37.657 INFO:teuthology.orchestra.run.smithi167.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T16:30:37.657 INFO:teuthology.orchestra.run.smithi167.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T16:30:37.657 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T16:30:37.657 INFO:teuthology.orchestra.run.smithi167.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T16:30:37.658 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T16:30:37.658 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T16:30:37.658 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T16:30:37.658 INFO:teuthology.orchestra.run.smithi167.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-25T16:30:37.658 INFO:teuthology.orchestra.run.smithi167.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-25T16:30:37.658 INFO:teuthology.orchestra.run.smithi167.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T16:30:37.658 INFO:teuthology.orchestra.run.smithi167.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T16:30:37.658 INFO:teuthology.orchestra.run.smithi167.stdout:Installing weak dependencies: 2024-03-25T16:30:37.658 INFO:teuthology.orchestra.run.smithi167.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-25T16:30:37.658 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:30:37.658 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-03-25T16:30:37.658 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T16:30:37.658 INFO:teuthology.orchestra.run.smithi167.stdout:Install 30 Packages 2024-03-25T16:30:37.659 INFO:teuthology.orchestra.run.smithi167.stdout:Upgrade 2 Packages 2024-03-25T16:30:37.659 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:30:37.659 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 58 M 2024-03-25T16:30:37.659 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-03-25T16:30:39.305 INFO:teuthology.orchestra.run.smithi155.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 375 kB/s | 5.1 MB 00:13 2024-03-25T16:30:39.563 INFO:teuthology.orchestra.run.smithi155.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 175 kB/s | 45 kB 00:00 2024-03-25T16:30:40.438 INFO:teuthology.orchestra.run.smithi155.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 150 kB/s | 131 kB 00:00 2024-03-25T16:30:41.255 INFO:teuthology.orchestra.run.smithi155.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 198 kB/s | 162 kB 00:00 2024-03-25T16:30:42.097 INFO:teuthology.orchestra.run.smithi155.stdout:(9/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 368 kB/s | 9.7 MB 00:27 2024-03-25T16:30:42.756 INFO:teuthology.orchestra.run.smithi155.stdout:(10/32): python3-rados-19.0.0-1439.g5dcb190d.el 214 kB/s | 320 kB 00:01 2024-03-25T16:30:43.014 INFO:teuthology.orchestra.run.smithi155.stdout:(11/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 325 kB/s | 298 kB 00:00 2024-03-25T16:30:43.123 INFO:teuthology.orchestra.run.smithi155.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 979 kB/s | 106 kB 00:00 2024-03-25T16:30:43.173 INFO:teuthology.orchestra.run.smithi155.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-25T16:30:43.207 INFO:teuthology.orchestra.run.smithi155.stdout:(14/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 220 kB/s | 99 kB 00:00 2024-03-25T16:30:43.232 INFO:teuthology.orchestra.run.smithi155.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 755 kB/s | 44 kB 00:00 2024-03-25T16:30:43.274 INFO:teuthology.orchestra.run.smithi155.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-03-25T16:30:43.299 INFO:teuthology.orchestra.run.smithi155.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.2 MB/s | 282 kB 00:00 2024-03-25T16:30:43.324 INFO:teuthology.orchestra.run.smithi155.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 832 kB/s | 41 kB 00:00 2024-03-25T16:30:43.475 INFO:teuthology.orchestra.run.smithi155.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 232 kB/s | 35 kB 00:00 2024-03-25T16:30:43.534 INFO:teuthology.orchestra.run.smithi155.stdout:(20/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 763 kB/s | 44 kB 00:00 2024-03-25T16:30:43.584 INFO:teuthology.orchestra.run.smithi155.stdout:(21/32): qatlib-23.11.0-1.el9.x86_64.rpm 763 kB/s | 216 kB 00:00 2024-03-25T16:30:43.792 INFO:teuthology.orchestra.run.smithi155.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.2 MB/s | 309 kB 00:00 2024-03-25T16:30:43.843 INFO:teuthology.orchestra.run.smithi155.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 521 kB/s | 26 kB 00:00 2024-03-25T16:30:43.893 INFO:teuthology.orchestra.run.smithi155.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 972 kB/s | 49 kB 00:00 2024-03-25T16:30:43.952 INFO:teuthology.orchestra.run.smithi155.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-25T16:30:44.027 INFO:teuthology.orchestra.run.smithi155.stdout:(26/32): libarrow-9.0.0-10.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2024-03-25T16:30:44.077 INFO:teuthology.orchestra.run.smithi155.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 6.6 MB/s | 838 kB 00:00 2024-03-25T16:30:44.102 INFO:teuthology.orchestra.run.smithi155.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-03-25T16:30:44.186 INFO:teuthology.orchestra.run.smithi155.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-03-25T16:30:49.981 INFO:teuthology.orchestra.run.smithi167.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 449 kB/s | 5.3 MB 00:12 2024-03-25T16:30:50.081 INFO:teuthology.orchestra.run.smithi167.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 252 kB/s | 25 kB 00:00 2024-03-25T16:30:51.698 INFO:teuthology.orchestra.run.smithi167.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 459 kB/s | 742 kB 00:01 2024-03-25T16:30:53.182 INFO:teuthology.orchestra.run.smithi167.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 336 kB/s | 498 kB 00:01 2024-03-25T16:30:54.136 INFO:teuthology.orchestra.run.smithi155.stdout:(30/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 322 kB/s | 3.1 MB 00:09 2024-03-25T16:30:55.923 INFO:teuthology.orchestra.run.smithi167.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 550 kB/s | 9.7 MB 00:18 2024-03-25T16:30:56.106 INFO:teuthology.orchestra.run.smithi167.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 246 kB/s | 45 kB 00:00 2024-03-25T16:30:56.260 INFO:teuthology.orchestra.run.smithi155.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 284 kB/s | 3.4 MB 00:12 2024-03-25T16:30:56.448 INFO:teuthology.orchestra.run.smithi167.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 384 kB/s | 131 kB 00:00 2024-03-25T16:30:56.782 INFO:teuthology.orchestra.run.smithi167.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 486 kB/s | 162 kB 00:00 2024-03-25T16:30:57.624 INFO:teuthology.orchestra.run.smithi167.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 381 kB/s | 320 kB 00:00 2024-03-25T16:30:58.591 INFO:teuthology.orchestra.run.smithi167.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 308 kB/s | 298 kB 00:00 2024-03-25T16:30:59.008 INFO:teuthology.orchestra.run.smithi167.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 238 kB/s | 99 kB 00:00 2024-03-25T16:30:59.125 INFO:teuthology.orchestra.run.smithi167.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 911 kB/s | 106 kB 00:00 2024-03-25T16:30:59.175 INFO:teuthology.orchestra.run.smithi167.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-25T16:30:59.209 INFO:teuthology.orchestra.run.smithi167.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-25T16:30:59.276 INFO:teuthology.orchestra.run.smithi167.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-03-25T16:30:59.318 INFO:teuthology.orchestra.run.smithi167.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-25T16:30:59.351 INFO:teuthology.orchestra.run.smithi167.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-03-25T16:30:59.576 INFO:teuthology.orchestra.run.smithi167.stdout:(18/32): qatlib-23.11.0-1.el9.x86_64.rpm 963 kB/s | 216 kB 00:00 2024-03-25T16:30:59.635 INFO:teuthology.orchestra.run.smithi167.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 598 kB/s | 35 kB 00:00 2024-03-25T16:30:59.694 INFO:teuthology.orchestra.run.smithi167.stdout:(20/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 764 kB/s | 44 kB 00:00 2024-03-25T16:31:00.052 INFO:teuthology.orchestra.run.smithi167.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 862 kB/s | 309 kB 00:00 2024-03-25T16:31:00.369 INFO:teuthology.orchestra.run.smithi167.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-03-25T16:31:00.453 INFO:teuthology.orchestra.run.smithi167.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 312 kB/s | 26 kB 00:00 2024-03-25T16:31:00.553 INFO:teuthology.orchestra.run.smithi167.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 487 kB/s | 49 kB 00:00 2024-03-25T16:31:00.637 INFO:teuthology.orchestra.run.smithi167.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 808 kB/s | 67 kB 00:00 2024-03-25T16:31:00.737 INFO:teuthology.orchestra.run.smithi167.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 8.2 MB/s | 838 kB 00:00 2024-03-25T16:31:00.821 INFO:teuthology.orchestra.run.smithi167.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.2 MB/s | 192 kB 00:00 2024-03-25T16:31:00.930 INFO:teuthology.orchestra.run.smithi167.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-03-25T16:31:03.385 INFO:teuthology.orchestra.run.smithi155.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 421 kB/s | 20 MB 00:48 2024-03-25T16:31:03.388 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:31:03.388 INFO:teuthology.orchestra.run.smithi155.stdout:Total 1.2 MB/s | 58 MB 00:48 2024-03-25T16:31:03.421 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-25T16:31:03.459 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-25T16:31:03.459 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-25T16:31:03.932 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-25T16:31:03.933 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-25T16:31:04.838 INFO:teuthology.orchestra.run.smithi167.stdout:(29/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 448 kB/s | 5.1 MB 00:11 2024-03-25T16:31:05.404 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-25T16:31:05.479 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T16:31:05.514 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T16:31:05.540 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T16:31:05.571 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:31:05.588 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:31:05.763 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:31:05.819 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:31:06.152 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T16:31:06.217 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:31:06.307 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:31:06.395 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T16:31:06.418 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:31:06.489 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:31:06.517 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T16:31:06.561 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T16:31:06.628 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:31:06.653 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:31:06.716 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:31:06.743 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-25T16:31:06.851 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T16:31:06.895 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T16:31:06.935 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T16:31:06.968 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T16:31:06.996 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T16:31:07.037 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T16:31:07.357 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T16:31:07.384 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:31:07.465 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:31:07.506 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T16:31:07.551 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T16:31:08.005 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T16:31:08.028 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:31:08.083 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:31:08.151 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T16:31:08.181 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T16:31:10.345 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:31:10.385 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:31:10.936 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:31:10.980 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:31:11.065 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:31:11.065 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T16:31:11.065 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T16:31:11.065 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:31:11.106 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:31:13.012 INFO:teuthology.orchestra.run.smithi167.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 286 kB/s | 3.4 MB 00:12 2024-03-25T16:31:19.328 INFO:teuthology.orchestra.run.smithi167.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 221 kB/s | 3.1 MB 00:14 2024-03-25T16:31:20.249 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:31:20.249 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /sys 2024-03-25T16:31:20.249 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /proc 2024-03-25T16:31:20.249 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /mnt 2024-03-25T16:31:20.249 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /var/tmp 2024-03-25T16:31:20.249 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /home 2024-03-25T16:31:20.249 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /root 2024-03-25T16:31:20.249 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /tmp 2024-03-25T16:31:20.250 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:31:21.175 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:31:21.200 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:31:21.200 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:31:21.200 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T16:31:21.200 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:31:21.200 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:31:21.201 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:31:21.202 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:31:21.295 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:31:21.295 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:31:22.451 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:31:22.451 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T16:31:22.451 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T16:31:22.451 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T16:31:22.451 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T16:31:22.451 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T16:31:22.451 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T16:31:22.451 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:31:22.451 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T16:31:22.451 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T16:31:22.451 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T16:31:22.452 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T16:31:22.452 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:31:22.452 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T16:31:22.452 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T16:31:22.452 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T16:31:22.452 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T16:31:22.452 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T16:31:22.452 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T16:31:22.452 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T16:31:22.452 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-25T16:31:22.452 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-25T16:31:22.452 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-25T16:31:22.452 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T16:31:22.452 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T16:31:22.452 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T16:31:22.453 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T16:31:22.453 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T16:31:22.453 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T16:31:22.453 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T16:31:22.453 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T16:31:22.453 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:31:22.453 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T16:31:22.453 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T16:31:23.270 INFO:teuthology.orchestra.run.smithi167.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 448 kB/s | 20 MB 00:45 2024-03-25T16:31:23.273 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:31:23.273 INFO:teuthology.orchestra.run.smithi167.stdout:Total 1.3 MB/s | 58 MB 00:45 2024-03-25T16:31:23.305 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-03-25T16:31:23.342 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-03-25T16:31:23.342 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-03-25T16:31:23.485 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:31:23.485 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:31:23.485 INFO:teuthology.orchestra.run.smithi155.stdout:Upgraded: 2024-03-25T16:31:23.485 INFO:teuthology.orchestra.run.smithi155.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:31:23.485 INFO:teuthology.orchestra.run.smithi155.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:31:23.485 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-25T16:31:23.485 INFO:teuthology.orchestra.run.smithi155.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T16:31:23.485 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:31:23.485 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:31:23.485 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:31:23.485 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:31:23.485 INFO:teuthology.orchestra.run.smithi155.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T16:31:23.486 INFO:teuthology.orchestra.run.smithi155.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T16:31:23.486 INFO:teuthology.orchestra.run.smithi155.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T16:31:23.486 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:31:23.486 INFO:teuthology.orchestra.run.smithi155.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T16:31:23.486 INFO:teuthology.orchestra.run.smithi155.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T16:31:23.486 INFO:teuthology.orchestra.run.smithi155.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T16:31:23.486 INFO:teuthology.orchestra.run.smithi155.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:31:23.486 INFO:teuthology.orchestra.run.smithi155.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T16:31:23.486 INFO:teuthology.orchestra.run.smithi155.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:31:23.486 INFO:teuthology.orchestra.run.smithi155.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T16:31:23.486 INFO:teuthology.orchestra.run.smithi155.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T16:31:23.486 INFO:teuthology.orchestra.run.smithi155.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T16:31:23.486 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:31:23.487 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:31:23.487 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:31:23.487 INFO:teuthology.orchestra.run.smithi155.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T16:31:23.487 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:31:23.487 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:31:23.487 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:31:23.487 INFO:teuthology.orchestra.run.smithi155.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-25T16:31:23.487 INFO:teuthology.orchestra.run.smithi155.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-25T16:31:23.487 INFO:teuthology.orchestra.run.smithi155.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-25T16:31:23.487 INFO:teuthology.orchestra.run.smithi155.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T16:31:23.487 INFO:teuthology.orchestra.run.smithi155.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T16:31:23.487 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:31:23.487 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-25T16:31:23.814 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-03-25T16:31:23.815 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-03-25T16:31:23.851 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-test 2024-03-25T16:31:24.408 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:12 ago on Mon 25 Mar 2024 04:30:12 PM UTC. 2024-03-25T16:31:24.555 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-25T16:31:24.556 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T16:31:24.556 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-03-25T16:31:24.556 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T16:31:24.556 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-25T16:31:24.556 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T16:31:24.557 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-03-25T16:31:24.557 INFO:teuthology.orchestra.run.smithi155.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T16:31:24.557 INFO:teuthology.orchestra.run.smithi155.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T16:31:24.557 INFO:teuthology.orchestra.run.smithi155.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T16:31:24.557 INFO:teuthology.orchestra.run.smithi155.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T16:31:24.557 INFO:teuthology.orchestra.run.smithi155.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T16:31:24.557 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:31:24.557 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-25T16:31:24.557 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T16:31:24.557 INFO:teuthology.orchestra.run.smithi155.stdout:Install 6 Packages 2024-03-25T16:31:24.557 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:31:24.558 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 48 M 2024-03-25T16:31:24.558 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 204 M 2024-03-25T16:31:24.558 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-25T16:31:24.765 INFO:teuthology.orchestra.run.smithi155.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-25T16:31:24.823 INFO:teuthology.orchestra.run.smithi155.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2024-03-25T16:31:24.866 INFO:teuthology.orchestra.run.smithi155.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-25T16:31:25.024 INFO:teuthology.orchestra.run.smithi155.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-25T16:31:25.272 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-03-25T16:31:25.372 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T16:31:25.408 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T16:31:25.439 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T16:31:25.464 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:31:25.481 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:31:25.508 INFO:teuthology.orchestra.run.smithi155.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 180 kB/s | 163 kB 00:00 2024-03-25T16:31:25.674 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:31:25.728 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:31:26.049 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T16:31:26.118 INFO:teuthology.orchestra.run.smithi167.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:31:26.211 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:31:26.297 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T16:31:26.320 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:31:26.399 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:31:26.428 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T16:31:26.472 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T16:31:26.548 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:31:26.572 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:31:26.618 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:31:26.645 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-25T16:31:26.754 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T16:31:26.798 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T16:31:26.838 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T16:31:26.870 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T16:31:26.899 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T16:31:26.932 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T16:31:27.241 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T16:31:27.270 INFO:teuthology.orchestra.run.smithi167.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:31:27.360 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:31:27.391 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T16:31:27.445 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T16:31:27.908 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T16:31:27.939 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:31:27.994 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:31:28.063 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T16:31:28.092 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T16:31:30.234 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:31:30.282 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:31:30.829 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:31:30.867 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:31:30.949 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:31:30.949 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T16:31:30.949 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T16:31:30.949 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:31:30.985 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:31:40.373 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:31:40.374 INFO:teuthology.orchestra.run.smithi167.stdout:skipping the directory /sys 2024-03-25T16:31:40.374 INFO:teuthology.orchestra.run.smithi167.stdout:skipping the directory /proc 2024-03-25T16:31:40.374 INFO:teuthology.orchestra.run.smithi167.stdout:skipping the directory /mnt 2024-03-25T16:31:40.374 INFO:teuthology.orchestra.run.smithi167.stdout:skipping the directory /var/tmp 2024-03-25T16:31:40.374 INFO:teuthology.orchestra.run.smithi167.stdout:skipping the directory /home 2024-03-25T16:31:40.374 INFO:teuthology.orchestra.run.smithi167.stdout:skipping the directory /root 2024-03-25T16:31:40.374 INFO:teuthology.orchestra.run.smithi167.stdout:skipping the directory /tmp 2024-03-25T16:31:40.374 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:31:41.290 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:31:41.320 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:31:41.321 INFO:teuthology.orchestra.run.smithi167.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:31:41.321 INFO:teuthology.orchestra.run.smithi167.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T16:31:41.321 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:31:41.321 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:31:41.321 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:31:41.322 INFO:teuthology.orchestra.run.smithi167.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:31:41.393 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:31:41.393 INFO:teuthology.orchestra.run.smithi167.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:31:42.551 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:31:42.551 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T16:31:42.552 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T16:31:42.552 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T16:31:42.552 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T16:31:42.552 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T16:31:42.552 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T16:31:42.552 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:31:42.552 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T16:31:42.552 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T16:31:42.552 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T16:31:42.552 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T16:31:42.552 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:31:42.552 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T16:31:42.552 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T16:31:42.552 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T16:31:42.553 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T16:31:42.553 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T16:31:42.553 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T16:31:42.553 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T16:31:42.553 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-25T16:31:42.553 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-25T16:31:42.553 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-25T16:31:42.553 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T16:31:42.553 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T16:31:42.553 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T16:31:42.553 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T16:31:42.553 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T16:31:42.553 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T16:31:42.553 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T16:31:42.554 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T16:31:42.554 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:31:42.554 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T16:31:42.554 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T16:31:43.734 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:31:43.735 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:31:43.735 INFO:teuthology.orchestra.run.smithi167.stdout:Upgraded: 2024-03-25T16:31:43.735 INFO:teuthology.orchestra.run.smithi167.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:31:43.735 INFO:teuthology.orchestra.run.smithi167.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:31:43.735 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-03-25T16:31:43.735 INFO:teuthology.orchestra.run.smithi167.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T16:31:43.735 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:31:43.735 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:31:43.735 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:31:43.735 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:31:43.735 INFO:teuthology.orchestra.run.smithi167.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T16:31:43.735 INFO:teuthology.orchestra.run.smithi167.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T16:31:43.736 INFO:teuthology.orchestra.run.smithi167.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T16:31:43.736 INFO:teuthology.orchestra.run.smithi167.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:31:43.736 INFO:teuthology.orchestra.run.smithi167.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T16:31:43.736 INFO:teuthology.orchestra.run.smithi167.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T16:31:43.736 INFO:teuthology.orchestra.run.smithi167.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T16:31:43.736 INFO:teuthology.orchestra.run.smithi167.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:31:43.736 INFO:teuthology.orchestra.run.smithi167.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T16:31:43.736 INFO:teuthology.orchestra.run.smithi167.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:31:43.736 INFO:teuthology.orchestra.run.smithi167.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T16:31:43.736 INFO:teuthology.orchestra.run.smithi167.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T16:31:43.736 INFO:teuthology.orchestra.run.smithi167.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T16:31:43.736 INFO:teuthology.orchestra.run.smithi167.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:31:43.736 INFO:teuthology.orchestra.run.smithi167.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:31:43.736 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:31:43.737 INFO:teuthology.orchestra.run.smithi167.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T16:31:43.737 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:31:43.737 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:31:43.737 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:31:43.737 INFO:teuthology.orchestra.run.smithi167.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-25T16:31:43.737 INFO:teuthology.orchestra.run.smithi167.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-25T16:31:43.737 INFO:teuthology.orchestra.run.smithi167.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-25T16:31:43.737 INFO:teuthology.orchestra.run.smithi167.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T16:31:43.737 INFO:teuthology.orchestra.run.smithi167.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T16:31:43.737 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:31:43.737 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-03-25T16:31:44.076 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-test 2024-03-25T16:31:44.642 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:09 ago on Mon 25 Mar 2024 04:30:35 PM UTC. 2024-03-25T16:31:44.791 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-03-25T16:31:44.792 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T16:31:44.792 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2024-03-25T16:31:44.792 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T16:31:44.792 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-03-25T16:31:44.792 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T16:31:44.792 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2024-03-25T16:31:44.793 INFO:teuthology.orchestra.run.smithi167.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T16:31:44.793 INFO:teuthology.orchestra.run.smithi167.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T16:31:44.793 INFO:teuthology.orchestra.run.smithi167.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T16:31:44.793 INFO:teuthology.orchestra.run.smithi167.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T16:31:44.793 INFO:teuthology.orchestra.run.smithi167.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T16:31:44.793 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:31:44.794 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-03-25T16:31:44.794 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T16:31:44.794 INFO:teuthology.orchestra.run.smithi167.stdout:Install 6 Packages 2024-03-25T16:31:44.794 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:31:44.795 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 48 M 2024-03-25T16:31:44.795 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 204 M 2024-03-25T16:31:44.796 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-03-25T16:31:45.074 INFO:teuthology.orchestra.run.smithi167.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.6 MB/s | 217 kB 00:00 2024-03-25T16:31:45.174 INFO:teuthology.orchestra.run.smithi167.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.5 MB/s | 300 kB 00:00 2024-03-25T16:31:45.200 INFO:teuthology.orchestra.run.smithi167.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.4 MB/s | 63 kB 00:00 2024-03-25T16:31:45.266 INFO:teuthology.orchestra.run.smithi167.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 2.8 MB/s | 187 kB 00:00 2024-03-25T16:31:46.217 INFO:teuthology.orchestra.run.smithi167.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 128 kB/s | 163 kB 00:01 2024-03-25T16:33:37.908 INFO:teuthology.orchestra.run.smithi155.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 363 kB/s | 47 MB 02:13 2024-03-25T16:33:37.909 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:33:37.910 INFO:teuthology.orchestra.run.smithi155.stdout:Total 370 kB/s | 48 MB 02:13 2024-03-25T16:33:37.920 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-25T16:33:37.940 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-25T16:33:37.940 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-25T16:33:38.328 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-25T16:33:38.363 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-25T16:33:38.906 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-25T16:33:38.973 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T16:33:39.015 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T16:33:39.055 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:33:39.099 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T16:33:39.135 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:33:43.586 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:33:43.659 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:33:46.035 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:33:46.035 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T16:33:46.035 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T16:33:46.035 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:33:46.035 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T16:33:46.035 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T16:33:46.403 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T16:33:46.404 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:33:46.404 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-25T16:33:46.404 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:33:46.404 INFO:teuthology.orchestra.run.smithi155.stdout: jq-1.6-16.el9.x86_64 2024-03-25T16:33:46.404 INFO:teuthology.orchestra.run.smithi155.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:33:46.404 INFO:teuthology.orchestra.run.smithi155.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T16:33:46.404 INFO:teuthology.orchestra.run.smithi155.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T16:33:46.404 INFO:teuthology.orchestra.run.smithi155.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T16:33:46.404 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:33:46.404 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-25T16:33:46.728 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph 2024-03-25T16:33:47.292 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:03:35 ago on Mon 25 Mar 2024 04:30:12 PM UTC. 2024-03-25T16:33:47.446 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-25T16:33:47.448 INFO:teuthology.orchestra.run.smithi155.stdout:========================================================================================= 2024-03-25T16:33:47.449 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-03-25T16:33:47.449 INFO:teuthology.orchestra.run.smithi155.stdout:========================================================================================= 2024-03-25T16:33:47.449 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-25T16:33:47.449 INFO:teuthology.orchestra.run.smithi155.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T16:33:47.449 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-03-25T16:33:47.449 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T16:33:47.449 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T16:33:47.449 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T16:33:47.449 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T16:33:47.449 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T16:33:47.449 INFO:teuthology.orchestra.run.smithi155.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T16:33:47.449 INFO:teuthology.orchestra.run.smithi155.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T16:33:47.450 INFO:teuthology.orchestra.run.smithi155.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T16:33:47.450 INFO:teuthology.orchestra.run.smithi155.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T16:33:47.450 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T16:33:47.450 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T16:33:47.450 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T16:33:47.450 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T16:33:47.450 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T16:33:47.450 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T16:33:47.450 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T16:33:47.450 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T16:33:47.450 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T16:33:47.450 INFO:teuthology.orchestra.run.smithi155.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T16:33:47.450 INFO:teuthology.orchestra.run.smithi155.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T16:33:47.451 INFO:teuthology.orchestra.run.smithi155.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T16:33:47.451 INFO:teuthology.orchestra.run.smithi155.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T16:33:47.451 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T16:33:47.451 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T16:33:47.451 INFO:teuthology.orchestra.run.smithi155.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T16:33:47.451 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T16:33:47.451 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T16:33:47.451 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T16:33:47.451 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T16:33:47.451 INFO:teuthology.orchestra.run.smithi155.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T16:33:47.451 INFO:teuthology.orchestra.run.smithi155.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T16:33:47.451 INFO:teuthology.orchestra.run.smithi155.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T16:33:47.451 INFO:teuthology.orchestra.run.smithi155.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T16:33:47.451 INFO:teuthology.orchestra.run.smithi155.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T16:33:47.452 INFO:teuthology.orchestra.run.smithi155.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T16:33:47.452 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:33:47.452 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-25T16:33:47.452 INFO:teuthology.orchestra.run.smithi155.stdout:========================================================================================= 2024-03-25T16:33:47.452 INFO:teuthology.orchestra.run.smithi155.stdout:Install 36 Packages 2024-03-25T16:33:47.452 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:33:47.452 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 30 M 2024-03-25T16:33:47.453 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 107 M 2024-03-25T16:33:47.453 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-25T16:33:48.354 INFO:teuthology.orchestra.run.smithi155.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 17 kB/s | 6.6 kB 00:00 2024-03-25T16:33:57.745 INFO:teuthology.orchestra.run.smithi155.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 156 kB/s | 1.5 MB 00:09 2024-03-25T16:33:59.270 INFO:teuthology.orchestra.run.smithi155.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 192 kB/s | 2.1 MB 00:11 2024-03-25T16:34:00.170 INFO:teuthology.orchestra.run.smithi155.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 275 kB/s | 247 kB 00:00 2024-03-25T16:34:00.378 INFO:teuthology.orchestra.run.smithi155.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 898 kB/s | 187 kB 00:00 2024-03-25T16:34:00.446 INFO:teuthology.orchestra.run.smithi155.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.1 MB/s | 144 kB 00:00 2024-03-25T16:34:00.529 INFO:teuthology.orchestra.run.smithi155.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 379 kB/s | 32 kB 00:00 2024-03-25T16:34:00.579 INFO:teuthology.orchestra.run.smithi155.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 945 kB/s | 47 kB 00:00 2024-03-25T16:34:00.621 INFO:teuthology.orchestra.run.smithi155.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 925 kB/s | 39 kB 00:00 2024-03-25T16:34:00.680 INFO:teuthology.orchestra.run.smithi155.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-03-25T16:34:00.789 INFO:teuthology.orchestra.run.smithi155.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 11 MB/s | 1.1 MB 00:00 2024-03-25T16:34:00.839 INFO:teuthology.orchestra.run.smithi155.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 2.0 MB/s | 103 kB 00:00 2024-03-25T16:34:00.889 INFO:teuthology.orchestra.run.smithi155.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-25T16:34:00.940 INFO:teuthology.orchestra.run.smithi155.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-25T16:34:01.007 INFO:teuthology.orchestra.run.smithi155.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 1.8 MB/s | 125 kB 00:00 2024-03-25T16:34:01.150 INFO:teuthology.orchestra.run.smithi155.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 157 kB/s | 22 kB 00:00 2024-03-25T16:34:01.691 INFO:teuthology.orchestra.run.smithi155.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 279 kB/s | 151 kB 00:00 2024-03-25T16:34:01.816 INFO:teuthology.orchestra.run.smithi155.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 348 kB/s | 43 kB 00:00 2024-03-25T16:34:01.991 INFO:teuthology.orchestra.run.smithi155.stdout:(19/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 352 kB/s | 4.7 MB 00:13 2024-03-25T16:34:02.016 INFO:teuthology.orchestra.run.smithi155.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 858 kB/s | 172 kB 00:00 2024-03-25T16:34:02.117 INFO:teuthology.orchestra.run.smithi155.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 107 kB/s | 11 kB 00:00 2024-03-25T16:34:02.208 INFO:teuthology.orchestra.run.smithi155.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 193 kB/s | 18 kB 00:00 2024-03-25T16:34:02.317 INFO:teuthology.orchestra.run.smithi155.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 213 kB/s | 23 kB 00:00 2024-03-25T16:34:02.418 INFO:teuthology.orchestra.run.smithi155.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 194 kB/s | 19 kB 00:00 2024-03-25T16:34:02.510 INFO:teuthology.orchestra.run.smithi155.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 215 kB/s | 20 kB 00:00 2024-03-25T16:34:02.618 INFO:teuthology.orchestra.run.smithi155.stdout:(26/36): python3-cherrypy-18.6.1-2.el9.noarch.r 572 kB/s | 358 kB 00:00 2024-03-25T16:34:02.635 INFO:teuthology.orchestra.run.smithi155.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 370 kB/s | 46 kB 00:00 2024-03-25T16:34:02.736 INFO:teuthology.orchestra.run.smithi155.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 675 kB/s | 79 kB 00:00 2024-03-25T16:34:02.836 INFO:teuthology.orchestra.run.smithi155.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.3 MB/s | 272 kB 00:00 2024-03-25T16:34:02.853 INFO:teuthology.orchestra.run.smithi155.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 141 kB/s | 16 kB 00:00 2024-03-25T16:34:02.953 INFO:teuthology.orchestra.run.smithi155.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 770 kB/s | 90 kB 00:00 2024-03-25T16:34:02.970 INFO:teuthology.orchestra.run.smithi155.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 306 kB/s | 36 kB 00:00 2024-03-25T16:34:03.120 INFO:teuthology.orchestra.run.smithi155.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.5 MB/s | 230 kB 00:00 2024-03-25T16:34:03.145 INFO:teuthology.orchestra.run.smithi155.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.4 MB/s | 427 kB 00:00 2024-03-25T16:34:03.213 INFO:teuthology.orchestra.run.smithi155.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 217 kB/s | 20 kB 00:00 2024-03-25T16:34:33.922 INFO:teuthology.orchestra.run.smithi167.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 286 kB/s | 47 MB 02:48 2024-03-25T16:34:33.924 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:34:33.924 INFO:teuthology.orchestra.run.smithi167.stdout:Total 292 kB/s | 48 MB 02:49 2024-03-25T16:34:33.936 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-03-25T16:34:33.955 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-03-25T16:34:33.955 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-03-25T16:34:34.337 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-03-25T16:34:34.337 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-03-25T16:34:34.928 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-03-25T16:34:35.014 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T16:34:35.061 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T16:34:35.102 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:34:35.153 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T16:34:35.183 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:34:39.731 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:34:39.833 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:34:42.190 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:34:42.190 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T16:34:42.190 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T16:34:42.190 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:34:42.190 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T16:34:42.190 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T16:34:42.623 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T16:34:42.623 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:34:42.623 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-03-25T16:34:42.623 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:42.623 INFO:teuthology.orchestra.run.smithi167.stdout: jq-1.6-16.el9.x86_64 2024-03-25T16:34:42.623 INFO:teuthology.orchestra.run.smithi167.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:42.623 INFO:teuthology.orchestra.run.smithi167.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T16:34:42.623 INFO:teuthology.orchestra.run.smithi167.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T16:34:42.623 INFO:teuthology.orchestra.run.smithi167.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T16:34:42.623 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:34:42.623 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-03-25T16:34:42.959 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph 2024-03-25T16:34:43.528 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:04:08 ago on Mon 25 Mar 2024 04:30:35 PM UTC. 2024-03-25T16:34:43.689 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-03-25T16:34:43.692 INFO:teuthology.orchestra.run.smithi167.stdout:========================================================================================= 2024-03-25T16:34:43.692 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2024-03-25T16:34:43.692 INFO:teuthology.orchestra.run.smithi167.stdout:========================================================================================= 2024-03-25T16:34:43.692 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-03-25T16:34:43.692 INFO:teuthology.orchestra.run.smithi167.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T16:34:43.692 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2024-03-25T16:34:43.692 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T16:34:43.693 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T16:34:43.693 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T16:34:43.693 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T16:34:43.693 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T16:34:43.693 INFO:teuthology.orchestra.run.smithi167.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T16:34:43.693 INFO:teuthology.orchestra.run.smithi167.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T16:34:43.693 INFO:teuthology.orchestra.run.smithi167.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T16:34:43.693 INFO:teuthology.orchestra.run.smithi167.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T16:34:43.693 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T16:34:43.693 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T16:34:43.693 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T16:34:43.693 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T16:34:43.693 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T16:34:43.694 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T16:34:43.694 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T16:34:43.694 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T16:34:43.694 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T16:34:43.694 INFO:teuthology.orchestra.run.smithi167.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T16:34:43.694 INFO:teuthology.orchestra.run.smithi167.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T16:34:43.694 INFO:teuthology.orchestra.run.smithi167.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T16:34:43.694 INFO:teuthology.orchestra.run.smithi167.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T16:34:43.694 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T16:34:43.694 INFO:teuthology.orchestra.run.smithi167.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T16:34:43.697 INFO:teuthology.orchestra.run.smithi167.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T16:34:43.697 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T16:34:43.697 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T16:34:43.698 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T16:34:43.698 INFO:teuthology.orchestra.run.smithi167.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T16:34:43.698 INFO:teuthology.orchestra.run.smithi167.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T16:34:43.698 INFO:teuthology.orchestra.run.smithi167.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T16:34:43.698 INFO:teuthology.orchestra.run.smithi167.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T16:34:43.698 INFO:teuthology.orchestra.run.smithi167.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T16:34:43.698 INFO:teuthology.orchestra.run.smithi167.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T16:34:43.698 INFO:teuthology.orchestra.run.smithi167.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T16:34:43.698 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:34:43.698 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-03-25T16:34:43.698 INFO:teuthology.orchestra.run.smithi167.stdout:========================================================================================= 2024-03-25T16:34:43.699 INFO:teuthology.orchestra.run.smithi167.stdout:Install 36 Packages 2024-03-25T16:34:43.699 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:34:43.699 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 30 M 2024-03-25T16:34:43.699 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 107 M 2024-03-25T16:34:43.699 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-03-25T16:34:44.878 INFO:teuthology.orchestra.run.smithi167.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 15 kB/s | 6.6 kB 00:00 2024-03-25T16:34:48.707 INFO:teuthology.orchestra.run.smithi155.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 337 kB/s | 17 MB 00:50 2024-03-25T16:34:48.713 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:34:48.713 INFO:teuthology.orchestra.run.smithi155.stdout:Total 496 kB/s | 30 MB 01:01 2024-03-25T16:34:48.736 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-25T16:34:48.779 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-25T16:34:48.779 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-25T16:34:49.180 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-25T16:34:49.180 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-25T16:34:50.028 INFO:teuthology.orchestra.run.smithi167.stdout:(2/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 389 kB/s | 2.1 MB 00:05 2024-03-25T16:34:50.504 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-25T16:34:50.576 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T16:34:50.615 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T16:34:50.645 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T16:34:50.717 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T16:34:50.753 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T16:34:50.794 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T16:34:50.885 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T16:34:50.972 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T16:34:51.013 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T16:34:51.085 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T16:34:51.147 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T16:34:51.194 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T16:34:51.251 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T16:34:51.306 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T16:34:51.378 INFO:teuthology.orchestra.run.smithi167.stdout:(3/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 220 kB/s | 1.5 MB 00:06 2024-03-25T16:34:51.491 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T16:34:51.557 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T16:34:51.596 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T16:34:51.630 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T16:34:51.661 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T16:34:51.738 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T16:34:51.771 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:34:51.804 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T16:34:51.853 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T16:34:51.888 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T16:34:51.923 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T16:34:52.012 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T16:34:52.054 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T16:34:52.136 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T16:34:52.288 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T16:34:52.414 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:34:52.579 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T16:34:52.609 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:34:52.634 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:34:52.634 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:34:52.635 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T16:34:52.635 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:34:52.635 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:34:52.635 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:34:52.779 INFO:teuthology.orchestra.run.smithi167.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 177 kB/s | 247 kB 00:01 2024-03-25T16:34:53.004 INFO:teuthology.orchestra.run.smithi167.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 830 kB/s | 187 kB 00:00 2024-03-25T16:34:53.079 INFO:teuthology.orchestra.run.smithi167.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.9 MB/s | 144 kB 00:00 2024-03-25T16:34:53.121 INFO:teuthology.orchestra.run.smithi167.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 758 kB/s | 32 kB 00:00 2024-03-25T16:34:53.163 INFO:teuthology.orchestra.run.smithi167.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-25T16:34:53.205 INFO:teuthology.orchestra.run.smithi167.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 928 kB/s | 39 kB 00:00 2024-03-25T16:34:53.280 INFO:teuthology.orchestra.run.smithi167.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.1 MB/s | 241 kB 00:00 2024-03-25T16:34:53.430 INFO:teuthology.orchestra.run.smithi167.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 7.6 MB/s | 1.1 MB 00:00 2024-03-25T16:34:53.498 INFO:teuthology.orchestra.run.smithi167.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 1.5 MB/s | 103 kB 00:00 2024-03-25T16:34:53.581 INFO:teuthology.orchestra.run.smithi167.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 1.5 MB/s | 124 kB 00:00 2024-03-25T16:34:53.748 INFO:teuthology.orchestra.run.smithi167.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 1.1 MB/s | 187 kB 00:00 2024-03-25T16:34:53.915 INFO:teuthology.orchestra.run.smithi167.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 749 kB/s | 125 kB 00:00 2024-03-25T16:34:54.085 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:34:54.111 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:34:54.112 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:34:54.112 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T16:34:54.112 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:34:54.112 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:34:54.112 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:34:54.115 INFO:teuthology.orchestra.run.smithi167.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 111 kB/s | 22 kB 00:00 2024-03-25T16:34:54.332 INFO:teuthology.orchestra.run.smithi167.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 698 kB/s | 151 kB 00:00 2024-03-25T16:34:54.391 INFO:teuthology.orchestra.run.smithi167.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 746 kB/s | 43 kB 00:00 2024-03-25T16:34:54.475 INFO:teuthology.orchestra.run.smithi167.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-03-25T16:34:54.553 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:34:54.567 INFO:teuthology.orchestra.run.smithi167.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.8 MB/s | 358 kB 00:00 2024-03-25T16:34:54.575 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:34:54.575 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:34:54.575 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T16:34:54.575 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:34:54.575 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:34:54.575 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:34:54.608 INFO:teuthology.orchestra.run.smithi167.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 257 kB/s | 11 kB 00:00 2024-03-25T16:34:54.651 INFO:teuthology.orchestra.run.smithi167.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 421 kB/s | 18 kB 00:00 2024-03-25T16:34:54.693 INFO:teuthology.orchestra.run.smithi167.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 556 kB/s | 23 kB 00:00 2024-03-25T16:34:54.735 INFO:teuthology.orchestra.run.smithi167.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 466 kB/s | 19 kB 00:00 2024-03-25T16:34:54.777 INFO:teuthology.orchestra.run.smithi167.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 472 kB/s | 20 kB 00:00 2024-03-25T16:34:54.787 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:34:54.809 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:34:54.809 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:34:54.809 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T16:34:54.809 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:34:54.809 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:34:54.810 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:34:54.836 INFO:teuthology.orchestra.run.smithi167.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 792 kB/s | 46 kB 00:00 2024-03-25T16:34:54.876 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:34:54.894 INFO:teuthology.orchestra.run.smithi167.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 1.3 MB/s | 79 kB 00:00 2024-03-25T16:34:54.970 INFO:teuthology.orchestra.run.smithi167.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-03-25T16:34:55.020 INFO:teuthology.orchestra.run.smithi167.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 329 kB/s | 16 kB 00:00 2024-03-25T16:34:55.078 INFO:teuthology.orchestra.run.smithi167.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2024-03-25T16:34:55.129 INFO:teuthology.orchestra.run.smithi167.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 715 kB/s | 36 kB 00:00 2024-03-25T16:34:55.187 INFO:teuthology.orchestra.run.smithi167.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.8 MB/s | 230 kB 00:00 2024-03-25T16:34:55.263 INFO:teuthology.orchestra.run.smithi167.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.6 MB/s | 427 kB 00:00 2024-03-25T16:34:55.313 INFO:teuthology.orchestra.run.smithi167.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 400 kB/s | 20 kB 00:00 2024-03-25T16:34:56.651 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:34:56.651 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T16:34:56.651 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T16:34:56.651 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T16:34:56.651 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T16:34:56.651 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T16:34:56.651 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T16:34:56.651 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T16:34:56.651 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T16:34:56.651 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T16:34:56.652 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T16:34:56.652 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T16:34:56.652 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T16:34:56.652 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T16:34:56.652 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T16:34:56.652 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T16:34:56.652 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T16:34:56.652 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T16:34:56.652 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T16:34:56.652 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T16:34:56.652 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T16:34:56.652 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:34:56.652 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T16:34:56.652 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T16:34:56.652 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T16:34:56.653 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T16:34:56.653 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T16:34:56.653 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T16:34:56.653 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T16:34:56.653 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T16:34:56.653 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:34:56.653 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T16:34:56.653 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T16:34:56.653 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T16:34:56.653 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T16:34:56.653 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T16:34:57.734 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T16:34:57.735 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:34:57.735 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-25T16:34:57.735 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:57.735 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:57.735 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:57.735 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:34:57.735 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:57.735 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:57.735 INFO:teuthology.orchestra.run.smithi155.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T16:34:57.735 INFO:teuthology.orchestra.run.smithi155.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T16:34:57.735 INFO:teuthology.orchestra.run.smithi155.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T16:34:57.735 INFO:teuthology.orchestra.run.smithi155.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T16:34:57.735 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T16:34:57.735 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T16:34:57.736 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T16:34:57.736 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T16:34:57.736 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T16:34:57.736 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T16:34:57.736 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T16:34:57.736 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T16:34:57.736 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T16:34:57.736 INFO:teuthology.orchestra.run.smithi155.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T16:34:57.736 INFO:teuthology.orchestra.run.smithi155.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T16:34:57.736 INFO:teuthology.orchestra.run.smithi155.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T16:34:57.736 INFO:teuthology.orchestra.run.smithi155.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T16:34:57.736 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T16:34:57.736 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T16:34:57.736 INFO:teuthology.orchestra.run.smithi155.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T16:34:57.737 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T16:34:57.737 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T16:34:57.737 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T16:34:57.737 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T16:34:57.737 INFO:teuthology.orchestra.run.smithi155.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T16:34:57.737 INFO:teuthology.orchestra.run.smithi155.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T16:34:57.737 INFO:teuthology.orchestra.run.smithi155.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T16:34:57.737 INFO:teuthology.orchestra.run.smithi155.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T16:34:57.737 INFO:teuthology.orchestra.run.smithi155.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T16:34:57.737 INFO:teuthology.orchestra.run.smithi155.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T16:34:57.737 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:34:57.737 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-25T16:34:58.100 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-base 2024-03-25T16:34:58.669 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:04:46 ago on Mon 25 Mar 2024 04:30:12 PM UTC. 2024-03-25T16:34:58.768 INFO:teuthology.orchestra.run.smithi155.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:34:58.819 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-25T16:34:58.820 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-03-25T16:34:58.820 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-25T16:34:58.883 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install cephadm 2024-03-25T16:34:59.445 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:04:47 ago on Mon 25 Mar 2024 04:30:12 PM UTC. 2024-03-25T16:34:59.593 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-25T16:34:59.594 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T16:34:59.594 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-03-25T16:34:59.594 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T16:34:59.594 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-25T16:34:59.594 INFO:teuthology.orchestra.run.smithi155.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T16:34:59.594 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:34:59.594 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-25T16:34:59.595 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T16:34:59.595 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-03-25T16:34:59.595 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:34:59.595 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 747 k 2024-03-25T16:34:59.595 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 754 k 2024-03-25T16:34:59.595 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-25T16:35:01.346 INFO:teuthology.orchestra.run.smithi167.stdout:(35/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 291 kB/s | 4.7 MB 00:16 2024-03-25T16:35:03.174 INFO:teuthology.orchestra.run.smithi155.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 209 kB/s | 747 kB 00:03 2024-03-25T16:35:03.175 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:35:03.175 INFO:teuthology.orchestra.run.smithi155.stdout:Total 209 kB/s | 747 kB 00:03 2024-03-25T16:35:03.175 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-25T16:35:03.181 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-25T16:35:03.181 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-25T16:35:03.187 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-25T16:35:03.188 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-25T16:35:03.267 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-25T16:35:03.711 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:35:03.795 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:35:04.188 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:35:04.462 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:35:04.463 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:35:04.463 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-25T16:35:04.463 INFO:teuthology.orchestra.run.smithi155.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:35:04.463 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:35:04.463 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-25T16:35:04.661 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T16:35:05.243 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:04:53 ago on Mon 25 Mar 2024 04:30:12 PM UTC. 2024-03-25T16:35:05.398 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-25T16:35:05.399 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T16:35:05.399 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-03-25T16:35:05.399 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T16:35:05.399 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-25T16:35:05.399 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T16:35:05.399 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:35:05.399 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-25T16:35:05.399 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T16:35:05.399 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-03-25T16:35:05.399 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:35:05.399 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 145 k 2024-03-25T16:35:05.399 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 436 k 2024-03-25T16:35:05.400 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-25T16:35:06.389 INFO:teuthology.orchestra.run.smithi155.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 147 kB/s | 145 kB 00:00 2024-03-25T16:35:06.390 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:35:06.390 INFO:teuthology.orchestra.run.smithi155.stdout:Total 147 kB/s | 145 kB 00:00 2024-03-25T16:35:06.390 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-25T16:35:06.399 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-25T16:35:06.400 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-25T16:35:06.456 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-25T16:35:06.456 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-25T16:35:06.607 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-25T16:35:06.665 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:35:06.689 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:35:06.689 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:35:06.689 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T16:35:06.690 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:35:07.645 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:35:07.646 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:35:07.646 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-25T16:35:07.646 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:35:07.646 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:35:07.646 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-25T16:35:07.909 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mgr 2024-03-25T16:35:08.486 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:04:56 ago on Mon 25 Mar 2024 04:30:12 PM UTC. 2024-03-25T16:35:08.585 INFO:teuthology.orchestra.run.smithi155.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:35:08.637 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-25T16:35:08.638 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-03-25T16:35:08.638 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-25T16:35:08.702 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T16:35:09.264 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:04:57 ago on Mon 25 Mar 2024 04:30:12 PM UTC. 2024-03-25T16:35:09.414 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-25T16:35:09.415 INFO:teuthology.orchestra.run.smithi155.stdout:====================================================================================== 2024-03-25T16:35:09.415 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-03-25T16:35:09.416 INFO:teuthology.orchestra.run.smithi155.stdout:====================================================================================== 2024-03-25T16:35:09.416 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-25T16:35:09.416 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T16:35:09.416 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-03-25T16:35:09.416 INFO:teuthology.orchestra.run.smithi155.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T16:35:09.416 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T16:35:09.416 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T16:35:09.416 INFO:teuthology.orchestra.run.smithi155.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T16:35:09.416 INFO:teuthology.orchestra.run.smithi155.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T16:35:09.416 INFO:teuthology.orchestra.run.smithi155.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T16:35:09.416 INFO:teuthology.orchestra.run.smithi155.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T16:35:09.417 INFO:teuthology.orchestra.run.smithi155.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T16:35:09.417 INFO:teuthology.orchestra.run.smithi155.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T16:35:09.417 INFO:teuthology.orchestra.run.smithi155.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T16:35:09.417 INFO:teuthology.orchestra.run.smithi155.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T16:35:09.417 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:35:09.417 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-25T16:35:09.417 INFO:teuthology.orchestra.run.smithi155.stdout:====================================================================================== 2024-03-25T16:35:09.417 INFO:teuthology.orchestra.run.smithi155.stdout:Install 12 Packages 2024-03-25T16:35:09.417 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:35:09.418 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 6.9 M 2024-03-25T16:35:09.418 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 26 M 2024-03-25T16:35:09.418 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-25T16:35:10.093 INFO:teuthology.orchestra.run.smithi155.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 32 kB/s | 15 kB 00:00 2024-03-25T16:35:10.244 INFO:teuthology.orchestra.run.smithi155.stdout:(2/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.7 MB/s | 1.0 MB 00:00 2024-03-25T16:35:10.269 INFO:teuthology.orchestra.run.smithi155.stdout:(3/12): ceph-grafana-dashboards-19.0.0-1439.g5d 48 kB/s | 31 kB 00:00 2024-03-25T16:35:10.303 INFO:teuthology.orchestra.run.smithi155.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 3.9 MB/s | 238 kB 00:00 2024-03-25T16:35:10.586 INFO:teuthology.orchestra.run.smithi155.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.7 MB/s | 863 kB 00:00 2024-03-25T16:35:10.628 INFO:teuthology.orchestra.run.smithi155.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.6 MB/s | 548 kB 00:00 2024-03-25T16:35:10.795 INFO:teuthology.orchestra.run.smithi155.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 93 kB/s | 19 kB 00:00 2024-03-25T16:35:10.829 INFO:teuthology.orchestra.run.smithi155.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 12 MB/s | 2.0 MB 00:00 2024-03-25T16:35:10.896 INFO:teuthology.orchestra.run.smithi155.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 461 kB/s | 31 kB 00:00 2024-03-25T16:35:10.929 INFO:teuthology.orchestra.run.smithi155.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 1.1 MB/s | 144 kB 00:00 2024-03-25T16:35:10.954 INFO:teuthology.orchestra.run.smithi155.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.2 MB/s | 188 kB 00:00 2024-03-25T16:35:16.304 INFO:teuthology.orchestra.run.smithi155.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 273 kB/s | 1.8 MB 00:06 2024-03-25T16:35:16.311 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:35:16.311 INFO:teuthology.orchestra.run.smithi155.stdout:Total 1.0 MB/s | 6.9 MB 00:06 2024-03-25T16:35:16.331 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-25T16:35:16.351 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-25T16:35:16.351 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-25T16:35:16.531 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-25T16:35:16.531 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-25T16:35:17.055 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-25T16:35:17.130 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T16:35:17.164 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T16:35:17.211 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T16:35:17.242 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T16:35:17.342 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T16:35:17.467 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T16:35:17.552 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T16:35:17.605 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T16:35:17.642 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:35:17.683 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T16:35:17.924 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T16:35:17.957 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:35:18.760 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:35:18.760 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T16:35:18.760 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T16:35:18.760 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T16:35:18.760 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T16:35:18.760 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T16:35:18.760 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T16:35:18.761 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T16:35:18.761 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T16:35:18.761 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:35:18.761 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T16:35:18.761 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T16:35:19.102 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T16:35:19.102 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:35:19.102 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-25T16:35:19.102 INFO:teuthology.orchestra.run.smithi155.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T16:35:19.102 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:35:19.102 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:35:19.103 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:35:19.103 INFO:teuthology.orchestra.run.smithi155.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T16:35:19.103 INFO:teuthology.orchestra.run.smithi155.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T16:35:19.103 INFO:teuthology.orchestra.run.smithi155.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T16:35:19.103 INFO:teuthology.orchestra.run.smithi155.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T16:35:19.103 INFO:teuthology.orchestra.run.smithi155.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T16:35:19.103 INFO:teuthology.orchestra.run.smithi155.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T16:35:19.103 INFO:teuthology.orchestra.run.smithi155.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T16:35:19.103 INFO:teuthology.orchestra.run.smithi155.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T16:35:19.103 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:35:19.103 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-25T16:35:19.358 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T16:35:19.922 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:05:07 ago on Mon 25 Mar 2024 04:30:12 PM UTC. 2024-03-25T16:35:20.074 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-25T16:35:20.075 INFO:teuthology.orchestra.run.smithi155.stdout:============================================================================================ 2024-03-25T16:35:20.075 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-03-25T16:35:20.075 INFO:teuthology.orchestra.run.smithi155.stdout:============================================================================================ 2024-03-25T16:35:20.075 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-25T16:35:20.076 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T16:35:20.076 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-03-25T16:35:20.076 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T16:35:20.076 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T16:35:20.076 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T16:35:20.076 INFO:teuthology.orchestra.run.smithi155.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T16:35:20.076 INFO:teuthology.orchestra.run.smithi155.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T16:35:20.076 INFO:teuthology.orchestra.run.smithi155.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T16:35:20.076 INFO:teuthology.orchestra.run.smithi155.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T16:35:20.076 INFO:teuthology.orchestra.run.smithi155.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T16:35:20.076 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T16:35:20.076 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T16:35:20.076 INFO:teuthology.orchestra.run.smithi155.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T16:35:20.077 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:35:20.077 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-25T16:35:20.077 INFO:teuthology.orchestra.run.smithi155.stdout:============================================================================================ 2024-03-25T16:35:20.077 INFO:teuthology.orchestra.run.smithi155.stdout:Install 12 Packages 2024-03-25T16:35:20.077 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:35:20.077 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 37 M 2024-03-25T16:35:20.077 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 209 M 2024-03-25T16:35:20.078 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-25T16:35:20.353 INFO:teuthology.orchestra.run.smithi155.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 287 kB/s | 32 kB 00:00 2024-03-25T16:35:20.412 INFO:teuthology.orchestra.run.smithi155.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 502 kB/s | 17 kB 00:00 2024-03-25T16:35:20.470 INFO:teuthology.orchestra.run.smithi155.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 585 kB/s | 34 kB 00:00 2024-03-25T16:35:20.646 INFO:teuthology.orchestra.run.smithi155.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 27 MB/s | 4.7 MB 00:00 2024-03-25T16:35:20.804 INFO:teuthology.orchestra.run.smithi155.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 30 MB/s | 4.8 MB 00:00 2024-03-25T16:35:20.854 INFO:teuthology.orchestra.run.smithi155.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.9 MB/s | 3.0 MB 00:00 2024-03-25T16:35:20.879 INFO:teuthology.orchestra.run.smithi155.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.8 MB/s | 217 kB 00:00 2024-03-25T16:35:21.238 INFO:teuthology.orchestra.run.smithi155.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 41 MB/s | 16 MB 00:00 2024-03-25T16:35:22.122 INFO:teuthology.orchestra.run.smithi155.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 227 kB/s | 195 kB 00:00 2024-03-25T16:35:22.372 INFO:teuthology.orchestra.run.smithi155.stdout:(10/12): python3-devel-3.9.17-2.el9.x86_64.rpm 983 kB/s | 245 kB 00:00 2024-03-25T16:35:22.689 INFO:teuthology.orchestra.run.smithi155.stdout:(11/12): libgfortran-11.4.1-3.el9.x86_64.rpm 444 kB/s | 803 kB 00:01 2024-03-25T16:35:47.961 INFO:teuthology.orchestra.run.smithi155.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 273 kB/s | 7.4 MB 00:27 2024-03-25T16:35:47.964 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:35:47.964 INFO:teuthology.orchestra.run.smithi155.stdout:Total 1.3 MB/s | 37 MB 00:27 2024-03-25T16:35:48.006 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-25T16:35:48.033 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-25T16:35:48.033 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-25T16:35:48.392 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-25T16:35:48.392 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-25T16:35:49.113 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-25T16:35:49.175 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T16:35:49.206 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T16:35:49.308 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:35:49.368 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T16:35:49.548 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:35:49.584 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:35:49.666 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T16:35:50.327 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T16:35:50.447 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T16:35:51.713 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T16:35:52.901 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T16:35:52.935 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:35:55.603 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:35:55.603 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T16:35:55.603 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T16:35:55.603 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:35:55.603 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T16:35:55.603 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:35:55.603 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:35:55.603 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T16:35:55.603 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T16:35:55.603 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T16:35:55.603 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T16:35:55.604 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T16:35:56.057 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T16:35:56.057 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:35:56.057 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-25T16:35:56.057 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:35:56.057 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T16:35:56.057 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T16:35:56.057 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T16:35:56.057 INFO:teuthology.orchestra.run.smithi155.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T16:35:56.057 INFO:teuthology.orchestra.run.smithi155.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T16:35:56.057 INFO:teuthology.orchestra.run.smithi155.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T16:35:56.057 INFO:teuthology.orchestra.run.smithi155.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T16:35:56.058 INFO:teuthology.orchestra.run.smithi155.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T16:35:56.058 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T16:35:56.058 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T16:35:56.058 INFO:teuthology.orchestra.run.smithi155.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T16:35:56.058 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:35:56.058 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-25T16:35:56.422 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mgr-rook 2024-03-25T16:35:56.996 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:05:44 ago on Mon 25 Mar 2024 04:30:12 PM UTC. 2024-03-25T16:35:57.145 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-25T16:35:57.146 INFO:teuthology.orchestra.run.smithi155.stdout:======================================================================================== 2024-03-25T16:35:57.146 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-03-25T16:35:57.147 INFO:teuthology.orchestra.run.smithi155.stdout:======================================================================================== 2024-03-25T16:35:57.147 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-25T16:35:57.147 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T16:35:57.147 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-03-25T16:35:57.147 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T16:35:57.147 INFO:teuthology.orchestra.run.smithi155.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T16:35:57.147 INFO:teuthology.orchestra.run.smithi155.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T16:35:57.147 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T16:35:57.147 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T16:35:57.147 INFO:teuthology.orchestra.run.smithi155.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T16:35:57.147 INFO:teuthology.orchestra.run.smithi155.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T16:35:57.148 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T16:35:57.148 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T16:35:57.148 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T16:35:57.148 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T16:35:57.148 INFO:teuthology.orchestra.run.smithi155.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T16:35:57.148 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:35:57.148 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-25T16:35:57.148 INFO:teuthology.orchestra.run.smithi155.stdout:======================================================================================== 2024-03-25T16:35:57.148 INFO:teuthology.orchestra.run.smithi155.stdout:Install 13 Packages 2024-03-25T16:35:57.148 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:35:57.149 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 2.0 M 2024-03-25T16:35:57.149 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 26 M 2024-03-25T16:35:57.149 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-25T16:35:57.436 INFO:teuthology.orchestra.run.smithi155.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 236 kB/s | 24 kB 00:00 2024-03-25T16:35:57.486 INFO:teuthology.orchestra.run.smithi155.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 118 kB/s | 18 kB 00:00 2024-03-25T16:35:57.537 INFO:teuthology.orchestra.run.smithi155.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.2 MB/s | 174 kB 00:00 2024-03-25T16:35:57.562 INFO:teuthology.orchestra.run.smithi155.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-25T16:35:57.587 INFO:teuthology.orchestra.run.smithi155.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-25T16:35:57.612 INFO:teuthology.orchestra.run.smithi155.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 872 kB/s | 43 kB 00:00 2024-03-25T16:35:57.830 INFO:teuthology.orchestra.run.smithi155.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 69 kB/s | 15 kB 00:00 2024-03-25T16:35:57.855 INFO:teuthology.orchestra.run.smithi155.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 120 kB/s | 32 kB 00:00 2024-03-25T16:35:57.897 INFO:teuthology.orchestra.run.smithi155.stdout:(9/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 93 kB/s | 52 kB 00:00 2024-03-25T16:35:58.097 INFO:teuthology.orchestra.run.smithi155.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 759 kB/s | 203 kB 00:00 2024-03-25T16:35:58.164 INFO:teuthology.orchestra.run.smithi155.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 221 kB/s | 59 kB 00:00 2024-03-25T16:35:58.189 INFO:teuthology.orchestra.run.smithi155.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 972 kB/s | 90 kB 00:00 2024-03-25T16:35:58.298 INFO:teuthology.orchestra.run.smithi155.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.3 MB/s | 1.0 MB 00:00 2024-03-25T16:35:58.300 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:35:58.300 INFO:teuthology.orchestra.run.smithi155.stdout:Total 1.8 MB/s | 2.0 MB 00:01 2024-03-25T16:35:58.300 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-25T16:35:58.314 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-25T16:35:58.314 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-25T16:35:58.406 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-25T16:35:58.407 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-25T16:35:58.919 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-25T16:35:59.086 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T16:35:59.140 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T16:35:59.197 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T16:35:59.251 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T16:35:59.292 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T16:35:59.394 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T16:35:59.505 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T16:35:59.560 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T16:36:00.285 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T16:36:00.341 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T16:36:00.375 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T16:36:00.423 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T16:36:00.453 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:36:01.345 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:36:01.345 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T16:36:01.345 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T16:36:01.345 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T16:36:01.345 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T16:36:01.345 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T16:36:01.345 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T16:36:01.345 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T16:36:01.345 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T16:36:01.345 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T16:36:01.345 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T16:36:01.346 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T16:36:01.346 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T16:36:01.710 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T16:36:01.710 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:36:01.710 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-25T16:36:01.710 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:36:01.710 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T16:36:01.710 INFO:teuthology.orchestra.run.smithi155.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T16:36:01.710 INFO:teuthology.orchestra.run.smithi155.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T16:36:01.710 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T16:36:01.710 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T16:36:01.710 INFO:teuthology.orchestra.run.smithi155.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T16:36:01.710 INFO:teuthology.orchestra.run.smithi155.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T16:36:01.710 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T16:36:01.711 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T16:36:01.711 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T16:36:01.711 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T16:36:01.711 INFO:teuthology.orchestra.run.smithi155.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T16:36:01.711 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:36:01.711 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-25T16:36:01.992 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T16:36:02.555 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:05:50 ago on Mon 25 Mar 2024 04:30:12 PM UTC. 2024-03-25T16:36:02.705 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-25T16:36:02.705 INFO:teuthology.orchestra.run.smithi155.stdout:======================================================================================== 2024-03-25T16:36:02.705 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-03-25T16:36:02.706 INFO:teuthology.orchestra.run.smithi155.stdout:======================================================================================== 2024-03-25T16:36:02.706 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-25T16:36:02.706 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T16:36:02.706 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-03-25T16:36:02.706 INFO:teuthology.orchestra.run.smithi155.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T16:36:02.706 INFO:teuthology.orchestra.run.smithi155.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T16:36:02.706 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T16:36:02.706 INFO:teuthology.orchestra.run.smithi155.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T16:36:02.706 INFO:teuthology.orchestra.run.smithi155.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T16:36:02.706 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:36:02.706 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-25T16:36:02.706 INFO:teuthology.orchestra.run.smithi155.stdout:======================================================================================== 2024-03-25T16:36:02.707 INFO:teuthology.orchestra.run.smithi155.stdout:Install 6 Packages 2024-03-25T16:36:02.707 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:36:02.707 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 6.8 M 2024-03-25T16:36:02.707 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 33 M 2024-03-25T16:36:02.707 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-25T16:36:03.093 INFO:teuthology.orchestra.run.smithi155.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.4 MB/s | 248 kB 00:00 2024-03-25T16:36:03.218 INFO:teuthology.orchestra.run.smithi155.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 5.8 MB 00:00 2024-03-25T16:36:03.552 INFO:teuthology.orchestra.run.smithi155.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 173 kB/s | 58 kB 00:00 2024-03-25T16:36:03.710 INFO:teuthology.orchestra.run.smithi155.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 322 kB/s | 51 kB 00:00 2024-03-25T16:36:03.777 INFO:teuthology.orchestra.run.smithi155.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 804 kB/s | 548 kB 00:00 2024-03-25T16:36:03.802 INFO:teuthology.orchestra.run.smithi155.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 156 kB/s | 138 kB 00:00 2024-03-25T16:36:03.804 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:36:03.804 INFO:teuthology.orchestra.run.smithi155.stdout:Total 6.2 MB/s | 6.8 MB 00:01 2024-03-25T16:36:03.817 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-25T16:36:03.828 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-25T16:36:03.828 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-25T16:36:03.897 INFO:teuthology.orchestra.run.smithi167.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 233 kB/s | 17 MB 01:13 2024-03-25T16:36:03.901 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:36:03.901 INFO:teuthology.orchestra.run.smithi167.stdout:Total 379 kB/s | 30 MB 01:20 2024-03-25T16:36:03.920 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-25T16:36:03.920 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-25T16:36:03.922 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-03-25T16:36:03.964 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-03-25T16:36:03.964 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-03-25T16:36:04.208 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-25T16:36:04.386 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T16:36:04.395 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-03-25T16:36:04.396 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-03-25T16:36:04.439 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T16:36:04.953 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T16:36:05.033 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T16:36:05.094 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T16:36:05.117 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:36:05.717 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-03-25T16:36:05.783 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T16:36:05.826 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T16:36:05.861 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T16:36:05.932 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T16:36:05.941 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:36:05.941 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T16:36:05.941 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T16:36:05.941 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T16:36:05.942 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T16:36:05.942 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T16:36:05.984 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T16:36:06.017 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T16:36:06.109 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T16:36:06.188 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T16:36:06.228 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T16:36:06.228 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:36:06.229 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-25T16:36:06.229 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:36:06.229 INFO:teuthology.orchestra.run.smithi155.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T16:36:06.229 INFO:teuthology.orchestra.run.smithi155.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T16:36:06.229 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T16:36:06.229 INFO:teuthology.orchestra.run.smithi155.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T16:36:06.229 INFO:teuthology.orchestra.run.smithi155.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T16:36:06.229 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:36:06.229 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-25T16:36:06.237 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T16:36:06.308 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T16:36:06.363 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T16:36:06.402 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T16:36:06.450 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T16:36:06.505 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T16:36:06.520 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-fuse 2024-03-25T16:36:06.689 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T16:36:06.740 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T16:36:06.801 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T16:36:06.838 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T16:36:06.868 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T16:36:06.943 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T16:36:06.978 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:36:07.012 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T16:36:07.052 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T16:36:07.088 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:05:55 ago on Mon 25 Mar 2024 04:30:12 PM UTC. 2024-03-25T16:36:07.096 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T16:36:07.139 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T16:36:07.228 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T16:36:07.239 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-25T16:36:07.239 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T16:36:07.239 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-03-25T16:36:07.240 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T16:36:07.240 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-25T16:36:07.240 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T16:36:07.240 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-03-25T16:36:07.240 INFO:teuthology.orchestra.run.smithi155.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T16:36:07.240 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:36:07.240 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-25T16:36:07.240 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T16:36:07.240 INFO:teuthology.orchestra.run.smithi155.stdout:Install 2 Packages 2024-03-25T16:36:07.240 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:36:07.241 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 964 k 2024-03-25T16:36:07.241 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 2.8 M 2024-03-25T16:36:07.241 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-25T16:36:07.269 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T16:36:07.341 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T16:36:07.478 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T16:36:07.480 INFO:teuthology.orchestra.run.smithi155.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 329 kB/s | 79 kB 00:00 2024-03-25T16:36:07.594 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:36:07.763 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T16:36:07.792 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:36:07.817 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:36:07.817 INFO:teuthology.orchestra.run.smithi167.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:36:07.817 INFO:teuthology.orchestra.run.smithi167.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T16:36:07.817 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:36:07.817 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:36:07.818 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:36:09.326 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:36:09.352 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:36:09.352 INFO:teuthology.orchestra.run.smithi167.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:36:09.352 INFO:teuthology.orchestra.run.smithi167.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T16:36:09.352 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:36:09.352 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:36:09.352 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:36:09.521 INFO:teuthology.orchestra.run.smithi155.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 388 kB/s | 885 kB 00:02 2024-03-25T16:36:09.523 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:36:09.523 INFO:teuthology.orchestra.run.smithi155.stdout:Total 422 kB/s | 964 kB 00:02 2024-03-25T16:36:09.523 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-25T16:36:09.546 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-25T16:36:09.546 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-25T16:36:09.622 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-25T16:36:09.623 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-25T16:36:09.793 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:36:09.804 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-25T16:36:09.816 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:36:09.816 INFO:teuthology.orchestra.run.smithi167.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:36:09.816 INFO:teuthology.orchestra.run.smithi167.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T16:36:09.816 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:36:09.816 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:36:09.816 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:36:09.972 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T16:36:10.035 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:36:10.054 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:36:10.058 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:36:10.058 INFO:teuthology.orchestra.run.smithi167.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:36:10.058 INFO:teuthology.orchestra.run.smithi167.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T16:36:10.058 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:36:10.058 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:36:10.058 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:36:10.125 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:36:10.745 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:36:10.745 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:36:11.055 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T16:36:11.055 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:36:11.056 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-25T16:36:11.056 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T16:36:11.056 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:36:11.056 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-25T16:36:11.350 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-volume 2024-03-25T16:36:11.853 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:36:11.853 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T16:36:11.854 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T16:36:11.854 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T16:36:11.854 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T16:36:11.854 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T16:36:11.854 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T16:36:11.854 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T16:36:11.854 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T16:36:11.854 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T16:36:11.854 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T16:36:11.854 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T16:36:11.854 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T16:36:11.854 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T16:36:11.854 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T16:36:11.855 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T16:36:11.855 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T16:36:11.855 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T16:36:11.855 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T16:36:11.855 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T16:36:11.855 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T16:36:11.855 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:36:11.855 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T16:36:11.855 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T16:36:11.855 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T16:36:11.855 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T16:36:11.856 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T16:36:11.856 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T16:36:11.856 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T16:36:11.857 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T16:36:11.857 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:36:11.857 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T16:36:11.857 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T16:36:11.857 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T16:36:11.857 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T16:36:11.857 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T16:36:11.926 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:05:59 ago on Mon 25 Mar 2024 04:30:12 PM UTC. 2024-03-25T16:36:12.078 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-25T16:36:12.079 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T16:36:12.079 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-03-25T16:36:12.079 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T16:36:12.079 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-25T16:36:12.079 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T16:36:12.079 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-03-25T16:36:12.079 INFO:teuthology.orchestra.run.smithi155.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T16:36:12.080 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:36:12.080 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-25T16:36:12.080 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T16:36:12.080 INFO:teuthology.orchestra.run.smithi155.stdout:Install 2 Packages 2024-03-25T16:36:12.080 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:36:12.080 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 332 k 2024-03-25T16:36:12.080 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 1.5 M 2024-03-25T16:36:12.080 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-25T16:36:12.187 INFO:teuthology.orchestra.run.smithi155.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 646 kB/s | 69 kB 00:00 2024-03-25T16:36:12.828 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T16:36:12.828 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:36:12.828 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-03-25T16:36:12.828 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:36:12.828 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:36:12.828 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:36:12.828 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:36:12.828 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:36:12.828 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:36:12.828 INFO:teuthology.orchestra.run.smithi167.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T16:36:12.829 INFO:teuthology.orchestra.run.smithi167.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T16:36:12.829 INFO:teuthology.orchestra.run.smithi167.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T16:36:12.829 INFO:teuthology.orchestra.run.smithi167.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T16:36:12.829 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T16:36:12.829 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T16:36:12.829 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T16:36:12.829 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T16:36:12.829 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T16:36:12.829 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T16:36:12.829 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T16:36:12.829 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T16:36:12.829 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T16:36:12.829 INFO:teuthology.orchestra.run.smithi167.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T16:36:12.829 INFO:teuthology.orchestra.run.smithi167.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T16:36:12.830 INFO:teuthology.orchestra.run.smithi167.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T16:36:12.830 INFO:teuthology.orchestra.run.smithi167.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T16:36:12.830 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T16:36:12.830 INFO:teuthology.orchestra.run.smithi167.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T16:36:12.830 INFO:teuthology.orchestra.run.smithi167.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T16:36:12.830 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T16:36:12.830 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T16:36:12.830 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T16:36:12.830 INFO:teuthology.orchestra.run.smithi167.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T16:36:12.830 INFO:teuthology.orchestra.run.smithi167.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T16:36:12.830 INFO:teuthology.orchestra.run.smithi167.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T16:36:12.830 INFO:teuthology.orchestra.run.smithi167.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T16:36:12.830 INFO:teuthology.orchestra.run.smithi167.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T16:36:12.830 INFO:teuthology.orchestra.run.smithi167.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T16:36:12.831 INFO:teuthology.orchestra.run.smithi167.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T16:36:12.831 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:36:12.831 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-03-25T16:36:13.190 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-base 2024-03-25T16:36:13.760 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:05:38 ago on Mon 25 Mar 2024 04:30:35 PM UTC. 2024-03-25T16:36:13.856 INFO:teuthology.orchestra.run.smithi167.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:36:13.908 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-03-25T16:36:13.909 INFO:teuthology.orchestra.run.smithi167.stdout:Nothing to do. 2024-03-25T16:36:13.909 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-03-25T16:36:13.978 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install cephadm 2024-03-25T16:36:14.540 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:05:39 ago on Mon 25 Mar 2024 04:30:35 PM UTC. 2024-03-25T16:36:14.688 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-03-25T16:36:14.689 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T16:36:14.689 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2024-03-25T16:36:14.689 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T16:36:14.689 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-03-25T16:36:14.689 INFO:teuthology.orchestra.run.smithi167.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T16:36:14.689 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:36:14.689 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-03-25T16:36:14.689 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T16:36:14.689 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-03-25T16:36:14.689 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:36:14.690 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 747 k 2024-03-25T16:36:14.690 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 754 k 2024-03-25T16:36:14.690 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-03-25T16:36:15.737 INFO:teuthology.orchestra.run.smithi155.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 72 kB/s | 264 kB 00:03 2024-03-25T16:36:15.739 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:36:15.739 INFO:teuthology.orchestra.run.smithi155.stdout:Total 91 kB/s | 332 kB 00:03 2024-03-25T16:36:15.739 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-25T16:36:15.749 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-25T16:36:15.749 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-25T16:36:15.795 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-25T16:36:15.795 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-25T16:36:15.942 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-25T16:36:16.132 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T16:36:16.164 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:36:16.180 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:36:16.180 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:36:16.180 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T16:36:16.181 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:36:16.736 INFO:teuthology.orchestra.run.smithi167.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 365 kB/s | 747 kB 00:02 2024-03-25T16:36:16.737 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:36:16.737 INFO:teuthology.orchestra.run.smithi167.stdout:Total 365 kB/s | 747 kB 00:02 2024-03-25T16:36:16.737 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-03-25T16:36:16.743 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-03-25T16:36:16.744 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-03-25T16:36:16.750 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-03-25T16:36:16.750 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-03-25T16:36:16.825 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-03-25T16:36:16.953 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T16:36:17.312 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T16:36:17.313 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:36:17.313 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-25T16:36:17.313 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:36:17.313 INFO:teuthology.orchestra.run.smithi155.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T16:36:17.313 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:36:17.313 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-25T16:36:17.316 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:36:17.408 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:36:17.672 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install librados-devel 2024-03-25T16:36:17.783 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:36:18.044 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:36:18.044 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:36:18.044 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-03-25T16:36:18.044 INFO:teuthology.orchestra.run.smithi167.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:36:18.044 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:36:18.044 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-03-25T16:36:18.246 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T16:36:18.252 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:06:06 ago on Mon 25 Mar 2024 04:30:12 PM UTC. 2024-03-25T16:36:18.401 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-25T16:36:18.402 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T16:36:18.402 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-03-25T16:36:18.402 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T16:36:18.402 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-25T16:36:18.402 INFO:teuthology.orchestra.run.smithi155.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T16:36:18.402 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:36:18.403 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-25T16:36:18.403 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T16:36:18.403 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-03-25T16:36:18.403 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:36:18.403 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 127 k 2024-03-25T16:36:18.403 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 456 k 2024-03-25T16:36:18.403 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-25T16:36:18.833 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:05:43 ago on Mon 25 Mar 2024 04:30:35 PM UTC. 2024-03-25T16:36:18.987 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-03-25T16:36:18.988 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T16:36:18.988 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-03-25T16:36:18.988 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T16:36:18.988 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-03-25T16:36:18.988 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T16:36:18.988 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:36:18.988 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-03-25T16:36:18.989 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T16:36:18.989 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-03-25T16:36:18.989 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:36:18.989 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 145 k 2024-03-25T16:36:18.989 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 436 k 2024-03-25T16:36:18.989 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-03-25T16:36:19.436 INFO:teuthology.orchestra.run.smithi155.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 123 kB/s | 127 kB 00:01 2024-03-25T16:36:19.542 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:36:19.542 INFO:teuthology.orchestra.run.smithi155.stdout:Total 122 kB/s | 127 kB 00:01 2024-03-25T16:36:19.542 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-25T16:36:19.542 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-25T16:36:19.542 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-25T16:36:19.542 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-25T16:36:19.542 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-25T16:36:19.644 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-25T16:36:19.809 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:36:19.886 INFO:teuthology.orchestra.run.smithi167.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 162 kB/s | 145 kB 00:00 2024-03-25T16:36:19.887 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:36:19.887 INFO:teuthology.orchestra.run.smithi167.stdout:Total 162 kB/s | 145 kB 00:00 2024-03-25T16:36:19.887 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-03-25T16:36:19.896 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-03-25T16:36:19.897 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-03-25T16:36:19.950 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-03-25T16:36:19.951 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-03-25T16:36:20.097 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-03-25T16:36:20.162 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:36:20.187 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:36:20.187 INFO:teuthology.orchestra.run.smithi167.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:36:20.187 INFO:teuthology.orchestra.run.smithi167.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T16:36:20.187 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:36:20.279 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:36:20.522 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:36:20.522 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:36:20.523 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-25T16:36:20.523 INFO:teuthology.orchestra.run.smithi155.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:36:20.523 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:36:20.523 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-25T16:36:20.770 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install libcephfs2 2024-03-25T16:36:21.110 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:36:21.110 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:36:21.110 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-03-25T16:36:21.110 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:36:21.110 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:36:21.110 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-03-25T16:36:21.342 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:06:09 ago on Mon 25 Mar 2024 04:30:12 PM UTC. 2024-03-25T16:36:21.387 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-mgr 2024-03-25T16:36:21.438 INFO:teuthology.orchestra.run.smithi155.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:36:21.494 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-25T16:36:21.495 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-03-25T16:36:21.495 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-25T16:36:21.568 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install libcephfs-devel 2024-03-25T16:36:21.961 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:05:46 ago on Mon 25 Mar 2024 04:30:35 PM UTC. 2024-03-25T16:36:22.060 INFO:teuthology.orchestra.run.smithi167.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:36:22.112 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-03-25T16:36:22.113 INFO:teuthology.orchestra.run.smithi167.stdout:Nothing to do. 2024-03-25T16:36:22.113 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-03-25T16:36:22.138 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:06:10 ago on Mon 25 Mar 2024 04:30:12 PM UTC. 2024-03-25T16:36:22.172 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T16:36:22.289 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-25T16:36:22.289 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T16:36:22.290 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-03-25T16:36:22.290 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T16:36:22.290 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-25T16:36:22.290 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T16:36:22.290 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:36:22.290 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-25T16:36:22.290 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T16:36:22.290 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-03-25T16:36:22.290 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:36:22.290 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 31 k 2024-03-25T16:36:22.290 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 140 k 2024-03-25T16:36:22.290 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-25T16:36:22.724 INFO:teuthology.orchestra.run.smithi155.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 72 kB/s | 31 kB 00:00 2024-03-25T16:36:22.724 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:36:22.725 INFO:teuthology.orchestra.run.smithi155.stdout:Total 72 kB/s | 31 kB 00:00 2024-03-25T16:36:22.725 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-25T16:36:22.728 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-25T16:36:22.728 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-25T16:36:22.734 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:05:47 ago on Mon 25 Mar 2024 04:30:35 PM UTC. 2024-03-25T16:36:22.754 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-25T16:36:22.755 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-25T16:36:22.880 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-25T16:36:22.884 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-03-25T16:36:22.886 INFO:teuthology.orchestra.run.smithi167.stdout:====================================================================================== 2024-03-25T16:36:22.886 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2024-03-25T16:36:22.886 INFO:teuthology.orchestra.run.smithi167.stdout:====================================================================================== 2024-03-25T16:36:22.886 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-03-25T16:36:22.886 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T16:36:22.887 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2024-03-25T16:36:22.887 INFO:teuthology.orchestra.run.smithi167.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T16:36:22.887 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T16:36:22.887 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T16:36:22.887 INFO:teuthology.orchestra.run.smithi167.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T16:36:22.888 INFO:teuthology.orchestra.run.smithi167.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T16:36:22.888 INFO:teuthology.orchestra.run.smithi167.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T16:36:22.888 INFO:teuthology.orchestra.run.smithi167.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T16:36:22.888 INFO:teuthology.orchestra.run.smithi167.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T16:36:22.888 INFO:teuthology.orchestra.run.smithi167.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T16:36:22.888 INFO:teuthology.orchestra.run.smithi167.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T16:36:22.889 INFO:teuthology.orchestra.run.smithi167.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T16:36:22.889 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:36:22.889 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-03-25T16:36:22.889 INFO:teuthology.orchestra.run.smithi167.stdout:====================================================================================== 2024-03-25T16:36:22.889 INFO:teuthology.orchestra.run.smithi167.stdout:Install 12 Packages 2024-03-25T16:36:22.889 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:36:22.891 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 6.9 M 2024-03-25T16:36:22.891 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 26 M 2024-03-25T16:36:22.891 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-03-25T16:36:22.980 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:36:23.374 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:36:23.646 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:36:23.646 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:36:23.646 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-25T16:36:23.646 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:36:23.646 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:36:23.646 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-25T16:36:23.741 INFO:teuthology.orchestra.run.smithi167.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 59 kB/s | 31 kB 00:00 2024-03-25T16:36:23.800 INFO:teuthology.orchestra.run.smithi167.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 26 kB/s | 15 kB 00:00 2024-03-25T16:36:23.809 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install librados2 2024-03-25T16:36:23.908 INFO:teuthology.orchestra.run.smithi167.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.0 MB/s | 1.0 MB 00:00 2024-03-25T16:36:23.933 INFO:teuthology.orchestra.run.smithi167.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.8 MB/s | 238 kB 00:00 2024-03-25T16:36:24.059 INFO:teuthology.orchestra.run.smithi167.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 4.3 MB/s | 548 kB 00:00 2024-03-25T16:36:24.092 INFO:teuthology.orchestra.run.smithi167.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 596 kB/s | 19 kB 00:00 2024-03-25T16:36:24.185 INFO:teuthology.orchestra.run.smithi167.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 22 MB/s | 2.0 MB 00:00 2024-03-25T16:36:24.218 INFO:teuthology.orchestra.run.smithi167.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 4.2 MB/s | 144 kB 00:00 2024-03-25T16:36:24.252 INFO:teuthology.orchestra.run.smithi167.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 927 kB/s | 31 kB 00:00 2024-03-25T16:36:24.294 INFO:teuthology.orchestra.run.smithi167.stdout:(10/12): protobuf-compiler-3.14.0-13.el9.x86_64 2.2 MB/s | 863 kB 00:00 2024-03-25T16:36:24.319 INFO:teuthology.orchestra.run.smithi167.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.8 MB/s | 188 kB 00:00 2024-03-25T16:36:24.397 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:06:12 ago on Mon 25 Mar 2024 04:30:12 PM UTC. 2024-03-25T16:36:24.508 INFO:teuthology.orchestra.run.smithi155.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:36:24.556 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-25T16:36:24.557 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-03-25T16:36:24.558 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-25T16:36:24.605 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install librbd1 2024-03-25T16:36:25.201 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:06:13 ago on Mon 25 Mar 2024 04:30:12 PM UTC. 2024-03-25T16:36:25.303 INFO:teuthology.orchestra.run.smithi155.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:36:25.357 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-25T16:36:25.358 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-03-25T16:36:25.358 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-25T16:36:25.403 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install python3-rados 2024-03-25T16:36:25.968 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:06:13 ago on Mon 25 Mar 2024 04:30:12 PM UTC. 2024-03-25T16:36:26.064 INFO:teuthology.orchestra.run.smithi155.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:36:26.116 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-25T16:36:26.117 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-03-25T16:36:26.117 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-25T16:36:26.161 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install python3-rgw 2024-03-25T16:36:26.727 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:06:14 ago on Mon 25 Mar 2024 04:30:12 PM UTC. 2024-03-25T16:36:26.824 INFO:teuthology.orchestra.run.smithi155.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:36:26.875 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-25T16:36:26.876 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-03-25T16:36:26.876 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-25T16:36:26.925 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install python3-cephfs 2024-03-25T16:36:27.490 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:06:15 ago on Mon 25 Mar 2024 04:30:12 PM UTC. 2024-03-25T16:36:27.587 INFO:teuthology.orchestra.run.smithi155.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:36:27.638 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-25T16:36:27.639 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-03-25T16:36:27.639 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-25T16:36:27.684 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install python3-rbd 2024-03-25T16:36:28.247 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:06:16 ago on Mon 25 Mar 2024 04:30:12 PM UTC. 2024-03-25T16:36:28.344 INFO:teuthology.orchestra.run.smithi155.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:36:28.395 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-25T16:36:28.396 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-03-25T16:36:28.396 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-25T16:36:28.440 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install rbd-fuse 2024-03-25T16:36:29.007 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:06:17 ago on Mon 25 Mar 2024 04:30:12 PM UTC. 2024-03-25T16:36:29.157 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-25T16:36:29.158 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T16:36:29.158 INFO:teuthology.orchestra.run.smithi155.stdout: Package Architecture Version Repository Size 2024-03-25T16:36:29.158 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T16:36:29.158 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-25T16:36:29.158 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T16:36:29.158 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:36:29.158 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-25T16:36:29.158 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T16:36:29.158 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-03-25T16:36:29.158 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:36:29.158 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 85 k 2024-03-25T16:36:29.158 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 231 k 2024-03-25T16:36:29.159 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-25T16:36:29.961 INFO:teuthology.orchestra.run.smithi155.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 106 kB/s | 85 kB 00:00 2024-03-25T16:36:29.961 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:36:29.961 INFO:teuthology.orchestra.run.smithi155.stdout:Total 106 kB/s | 85 kB 00:00 2024-03-25T16:36:29.961 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-25T16:36:29.970 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-25T16:36:29.970 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-25T16:36:30.033 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-25T16:36:30.033 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-25T16:36:30.169 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-25T16:36:30.259 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:36:30.662 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:36:30.941 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:36:30.941 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:36:30.941 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-25T16:36:30.941 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:36:30.941 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:36:30.941 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-25T16:36:31.223 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install rbd-mirror 2024-03-25T16:36:31.798 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:06:19 ago on Mon 25 Mar 2024 04:30:12 PM UTC. 2024-03-25T16:36:31.951 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-25T16:36:31.952 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T16:36:31.952 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-03-25T16:36:31.952 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T16:36:31.952 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-25T16:36:31.952 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:36:31.952 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:36:31.952 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-25T16:36:31.952 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T16:36:31.952 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-03-25T16:36:31.953 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:36:31.953 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 3.1 M 2024-03-25T16:36:31.953 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 13 M 2024-03-25T16:36:31.953 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-25T16:36:32.760 INFO:teuthology.orchestra.run.smithi167.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 191 kB/s | 1.8 MB 00:09 2024-03-25T16:36:32.765 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:36:32.765 INFO:teuthology.orchestra.run.smithi167.stdout:Total 711 kB/s | 6.9 MB 00:09 2024-03-25T16:36:32.785 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-03-25T16:36:32.804 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-03-25T16:36:32.805 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-03-25T16:36:32.980 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-03-25T16:36:32.980 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-03-25T16:36:33.494 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-03-25T16:36:33.577 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T16:36:33.612 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T16:36:33.659 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T16:36:33.690 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T16:36:33.794 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T16:36:33.925 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T16:36:34.018 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T16:36:34.070 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T16:36:34.107 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:36:34.138 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T16:36:34.383 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T16:36:34.414 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:36:35.290 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:36:35.290 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T16:36:35.290 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T16:36:35.291 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T16:36:35.291 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T16:36:35.291 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T16:36:35.291 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T16:36:35.291 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T16:36:35.291 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T16:36:35.291 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:36:35.291 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T16:36:35.291 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T16:36:35.643 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T16:36:35.643 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:36:35.643 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-03-25T16:36:35.643 INFO:teuthology.orchestra.run.smithi167.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T16:36:35.643 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:36:35.643 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:36:35.643 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:36:35.643 INFO:teuthology.orchestra.run.smithi167.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T16:36:35.643 INFO:teuthology.orchestra.run.smithi167.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T16:36:35.644 INFO:teuthology.orchestra.run.smithi167.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T16:36:35.644 INFO:teuthology.orchestra.run.smithi167.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T16:36:35.644 INFO:teuthology.orchestra.run.smithi167.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T16:36:35.644 INFO:teuthology.orchestra.run.smithi167.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T16:36:35.644 INFO:teuthology.orchestra.run.smithi167.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T16:36:35.644 INFO:teuthology.orchestra.run.smithi167.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T16:36:35.644 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:36:35.644 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-03-25T16:36:35.912 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T16:36:36.477 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:06:01 ago on Mon 25 Mar 2024 04:30:35 PM UTC. 2024-03-25T16:36:36.628 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-03-25T16:36:36.629 INFO:teuthology.orchestra.run.smithi167.stdout:============================================================================================ 2024-03-25T16:36:36.629 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2024-03-25T16:36:36.629 INFO:teuthology.orchestra.run.smithi167.stdout:============================================================================================ 2024-03-25T16:36:36.630 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-03-25T16:36:36.630 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T16:36:36.630 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2024-03-25T16:36:36.630 INFO:teuthology.orchestra.run.smithi167.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T16:36:36.630 INFO:teuthology.orchestra.run.smithi167.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T16:36:36.630 INFO:teuthology.orchestra.run.smithi167.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T16:36:36.630 INFO:teuthology.orchestra.run.smithi167.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T16:36:36.630 INFO:teuthology.orchestra.run.smithi167.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T16:36:36.630 INFO:teuthology.orchestra.run.smithi167.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T16:36:36.630 INFO:teuthology.orchestra.run.smithi167.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T16:36:36.630 INFO:teuthology.orchestra.run.smithi167.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T16:36:36.630 INFO:teuthology.orchestra.run.smithi167.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T16:36:36.630 INFO:teuthology.orchestra.run.smithi167.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T16:36:36.631 INFO:teuthology.orchestra.run.smithi167.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T16:36:36.631 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:36:36.631 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-03-25T16:36:36.631 INFO:teuthology.orchestra.run.smithi167.stdout:============================================================================================ 2024-03-25T16:36:36.631 INFO:teuthology.orchestra.run.smithi167.stdout:Install 12 Packages 2024-03-25T16:36:36.631 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:36:36.631 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 37 M 2024-03-25T16:36:36.631 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 209 M 2024-03-25T16:36:36.632 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-03-25T16:36:37.087 INFO:teuthology.orchestra.run.smithi167.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 226 kB/s | 32 kB 00:00 2024-03-25T16:36:37.121 INFO:teuthology.orchestra.run.smithi167.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 523 kB/s | 17 kB 00:00 2024-03-25T16:36:37.204 INFO:teuthology.orchestra.run.smithi167.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-03-25T16:36:37.229 INFO:teuthology.orchestra.run.smithi167.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 319 kB/s | 34 kB 00:00 2024-03-25T16:36:37.438 INFO:teuthology.orchestra.run.smithi167.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 20 MB/s | 4.7 MB 00:00 2024-03-25T16:36:37.505 INFO:teuthology.orchestra.run.smithi167.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 17 MB/s | 4.8 MB 00:00 2024-03-25T16:36:37.530 INFO:teuthology.orchestra.run.smithi167.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-25T16:36:37.873 INFO:teuthology.orchestra.run.smithi167.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 43 MB/s | 16 MB 00:00 2024-03-25T16:36:38.006 INFO:teuthology.orchestra.run.smithi167.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.7 MB/s | 803 kB 00:00 2024-03-25T16:36:38.132 INFO:teuthology.orchestra.run.smithi167.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 756 kB/s | 195 kB 00:00 2024-03-25T16:36:38.540 INFO:teuthology.orchestra.run.smithi167.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 460 kB/s | 245 kB 00:00 2024-03-25T16:36:42.163 INFO:teuthology.orchestra.run.smithi155.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 310 kB/s | 3.1 MB 00:10 2024-03-25T16:36:42.164 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:36:42.164 INFO:teuthology.orchestra.run.smithi155.stdout:Total 310 kB/s | 3.1 MB 00:10 2024-03-25T16:36:42.165 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-25T16:36:42.176 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-25T16:36:42.176 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-25T16:36:42.252 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-25T16:36:42.253 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-25T16:36:42.673 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-25T16:36:42.739 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:36:42.763 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:36:42.763 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:36:42.763 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T16:36:42.764 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:36:42.764 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:36:42.764 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:36:43.822 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:36:43.822 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:36:43.822 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-25T16:36:43.822 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:36:43.822 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:36:43.822 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-25T16:36:44.066 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install rbd-nbd 2024-03-25T16:36:44.641 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:06:32 ago on Mon 25 Mar 2024 04:30:12 PM UTC. 2024-03-25T16:36:44.789 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-25T16:36:44.790 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T16:36:44.790 INFO:teuthology.orchestra.run.smithi155.stdout: Package Architecture Version Repository Size 2024-03-25T16:36:44.790 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T16:36:44.790 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-25T16:36:44.790 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T16:36:44.790 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:36:44.790 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-25T16:36:44.790 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T16:36:44.790 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-03-25T16:36:44.790 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:36:44.791 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 171 k 2024-03-25T16:36:44.791 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 494 k 2024-03-25T16:36:44.791 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-25T16:36:46.067 INFO:teuthology.orchestra.run.smithi155.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 134 kB/s | 171 kB 00:01 2024-03-25T16:36:46.068 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:36:46.068 INFO:teuthology.orchestra.run.smithi155.stdout:Total 134 kB/s | 171 kB 00:01 2024-03-25T16:36:46.069 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-25T16:36:46.078 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-25T16:36:46.078 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-25T16:36:46.138 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-25T16:36:46.138 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-25T16:36:46.290 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-25T16:36:46.374 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:36:46.867 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:36:47.148 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:36:47.148 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:36:47.148 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-25T16:36:47.148 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:36:47.148 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:36:47.148 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-25T16:36:47.371 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-base-debuginfo 2024-03-25T16:36:47.947 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:06:35 ago on Mon 25 Mar 2024 04:30:12 PM UTC. 2024-03-25T16:36:48.103 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-25T16:36:48.103 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T16:36:48.103 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-03-25T16:36:48.103 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T16:36:48.103 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-25T16:36:48.103 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T16:36:48.103 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-03-25T16:36:48.104 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T16:36:48.104 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:36:48.104 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-25T16:36:48.104 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T16:36:48.104 INFO:teuthology.orchestra.run.smithi155.stdout:Install 2 Packages 2024-03-25T16:36:48.104 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:36:48.104 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 91 M 2024-03-25T16:36:48.105 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 498 M 2024-03-25T16:36:48.105 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-25T16:36:54.488 INFO:teuthology.orchestra.run.smithi167.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 431 kB/s | 7.4 MB 00:17 2024-03-25T16:36:54.491 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:36:54.491 INFO:teuthology.orchestra.run.smithi167.stdout:Total 2.1 MB/s | 37 MB 00:17 2024-03-25T16:36:54.536 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-03-25T16:36:54.560 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-03-25T16:36:54.560 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-03-25T16:36:54.913 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-03-25T16:36:54.913 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-03-25T16:36:55.648 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-03-25T16:36:55.720 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T16:36:55.751 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T16:36:55.844 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:36:55.896 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T16:36:56.083 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:36:56.120 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:36:56.204 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T16:36:56.868 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T16:36:56.985 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T16:36:58.234 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T16:36:59.282 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T16:36:59.314 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:37:01.854 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:37:01.854 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T16:37:01.854 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T16:37:01.854 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:37:01.854 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T16:37:01.854 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:37:01.855 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:37:01.855 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T16:37:01.855 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T16:37:01.855 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T16:37:01.855 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T16:37:01.855 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T16:37:02.357 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T16:37:02.358 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:37:02.358 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-03-25T16:37:02.358 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:37:02.358 INFO:teuthology.orchestra.run.smithi167.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T16:37:02.358 INFO:teuthology.orchestra.run.smithi167.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T16:37:02.358 INFO:teuthology.orchestra.run.smithi167.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T16:37:02.358 INFO:teuthology.orchestra.run.smithi167.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T16:37:02.358 INFO:teuthology.orchestra.run.smithi167.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T16:37:02.358 INFO:teuthology.orchestra.run.smithi167.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T16:37:02.358 INFO:teuthology.orchestra.run.smithi167.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T16:37:02.358 INFO:teuthology.orchestra.run.smithi167.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T16:37:02.359 INFO:teuthology.orchestra.run.smithi167.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T16:37:02.359 INFO:teuthology.orchestra.run.smithi167.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T16:37:02.359 INFO:teuthology.orchestra.run.smithi167.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T16:37:02.359 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:37:02.359 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-03-25T16:37:02.841 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-mgr-rook 2024-03-25T16:37:03.421 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:06:28 ago on Mon 25 Mar 2024 04:30:35 PM UTC. 2024-03-25T16:37:03.571 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-03-25T16:37:03.572 INFO:teuthology.orchestra.run.smithi167.stdout:======================================================================================== 2024-03-25T16:37:03.573 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2024-03-25T16:37:03.573 INFO:teuthology.orchestra.run.smithi167.stdout:======================================================================================== 2024-03-25T16:37:03.573 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-03-25T16:37:03.573 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T16:37:03.573 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2024-03-25T16:37:03.573 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T16:37:03.573 INFO:teuthology.orchestra.run.smithi167.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T16:37:03.573 INFO:teuthology.orchestra.run.smithi167.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T16:37:03.573 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T16:37:03.573 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T16:37:03.573 INFO:teuthology.orchestra.run.smithi167.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T16:37:03.573 INFO:teuthology.orchestra.run.smithi167.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T16:37:03.573 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T16:37:03.574 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T16:37:03.574 INFO:teuthology.orchestra.run.smithi167.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T16:37:03.574 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T16:37:03.574 INFO:teuthology.orchestra.run.smithi167.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T16:37:03.574 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:37:03.574 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-03-25T16:37:03.574 INFO:teuthology.orchestra.run.smithi167.stdout:======================================================================================== 2024-03-25T16:37:03.574 INFO:teuthology.orchestra.run.smithi167.stdout:Install 13 Packages 2024-03-25T16:37:03.574 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:37:03.575 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 2.0 M 2024-03-25T16:37:03.575 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 26 M 2024-03-25T16:37:03.575 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-03-25T16:37:04.025 INFO:teuthology.orchestra.run.smithi167.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 253 kB/s | 24 kB 00:00 2024-03-25T16:37:04.108 INFO:teuthology.orchestra.run.smithi167.stdout:(2/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.9 MB/s | 174 kB 00:00 2024-03-25T16:37:04.150 INFO:teuthology.orchestra.run.smithi167.stdout:(3/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 132 kB 00:00 2024-03-25T16:37:04.192 INFO:teuthology.orchestra.run.smithi167.stdout:(4/13): python3-pyasn1-modules-0.4.8-6.el9.noar 5.0 MB/s | 211 kB 00:00 2024-03-25T16:37:04.217 INFO:teuthology.orchestra.run.smithi167.stdout:(5/13): python3-jsonpointer-2.0-4.el9.noarch.rp 62 kB/s | 18 kB 00:00 2024-03-25T16:37:04.243 INFO:teuthology.orchestra.run.smithi167.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 852 kB/s | 43 kB 00:00 2024-03-25T16:37:04.335 INFO:teuthology.orchestra.run.smithi167.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 163 kB/s | 15 kB 00:00 2024-03-25T16:37:04.360 INFO:teuthology.orchestra.run.smithi167.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 227 kB/s | 32 kB 00:00 2024-03-25T16:37:04.452 INFO:teuthology.orchestra.run.smithi167.stdout:(9/13): python3-google-auth-2.28.2-1.el9.noarch 1.7 MB/s | 203 kB 00:00 2024-03-25T16:37:04.502 INFO:teuthology.orchestra.run.smithi167.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-03-25T16:37:04.544 INFO:teuthology.orchestra.run.smithi167.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 5.5 MB/s | 1.0 MB 00:00 2024-03-25T16:37:04.569 INFO:teuthology.orchestra.run.smithi167.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.3 MB/s | 90 kB 00:00 2024-03-25T16:37:04.661 INFO:teuthology.orchestra.run.smithi167.stdout:(13/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el 71 kB/s | 52 kB 00:00 2024-03-25T16:37:04.663 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:37:04.663 INFO:teuthology.orchestra.run.smithi167.stdout:Total 1.9 MB/s | 2.0 MB 00:01 2024-03-25T16:37:04.663 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-03-25T16:37:04.676 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-03-25T16:37:04.676 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-03-25T16:37:04.769 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-03-25T16:37:04.771 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-03-25T16:37:05.283 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-03-25T16:37:05.450 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T16:37:05.504 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T16:37:05.568 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T16:37:05.614 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T16:37:05.663 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T16:37:05.765 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T16:37:05.875 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T16:37:05.933 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T16:37:06.666 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T16:37:06.713 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T16:37:06.746 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T16:37:06.804 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T16:37:06.825 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:37:07.653 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:37:07.653 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T16:37:07.653 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T16:37:07.653 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T16:37:07.653 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T16:37:07.653 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T16:37:07.654 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T16:37:07.654 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T16:37:07.654 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T16:37:07.654 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T16:37:07.654 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T16:37:07.654 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T16:37:07.654 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T16:37:08.005 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T16:37:08.005 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:37:08.005 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-03-25T16:37:08.005 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:37:08.005 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T16:37:08.005 INFO:teuthology.orchestra.run.smithi167.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T16:37:08.005 INFO:teuthology.orchestra.run.smithi167.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T16:37:08.005 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T16:37:08.005 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T16:37:08.006 INFO:teuthology.orchestra.run.smithi167.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T16:37:08.006 INFO:teuthology.orchestra.run.smithi167.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T16:37:08.006 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T16:37:08.006 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T16:37:08.006 INFO:teuthology.orchestra.run.smithi167.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T16:37:08.006 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T16:37:08.006 INFO:teuthology.orchestra.run.smithi167.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T16:37:08.006 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:37:08.006 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-03-25T16:37:08.258 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T16:37:08.826 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:06:33 ago on Mon 25 Mar 2024 04:30:35 PM UTC. 2024-03-25T16:37:08.976 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-03-25T16:37:08.977 INFO:teuthology.orchestra.run.smithi167.stdout:======================================================================================== 2024-03-25T16:37:08.977 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2024-03-25T16:37:08.977 INFO:teuthology.orchestra.run.smithi167.stdout:======================================================================================== 2024-03-25T16:37:08.977 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-03-25T16:37:08.977 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T16:37:08.977 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2024-03-25T16:37:08.977 INFO:teuthology.orchestra.run.smithi167.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T16:37:08.977 INFO:teuthology.orchestra.run.smithi167.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T16:37:08.978 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T16:37:08.978 INFO:teuthology.orchestra.run.smithi167.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T16:37:08.978 INFO:teuthology.orchestra.run.smithi167.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T16:37:08.978 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:37:08.978 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-03-25T16:37:08.978 INFO:teuthology.orchestra.run.smithi167.stdout:======================================================================================== 2024-03-25T16:37:08.978 INFO:teuthology.orchestra.run.smithi167.stdout:Install 6 Packages 2024-03-25T16:37:08.978 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:37:08.979 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 6.8 M 2024-03-25T16:37:08.979 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 33 M 2024-03-25T16:37:08.979 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-03-25T16:37:09.675 INFO:teuthology.orchestra.run.smithi167.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 5.8 MB 00:00 2024-03-25T16:37:09.916 INFO:teuthology.orchestra.run.smithi167.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 471 kB/s | 248 kB 00:00 2024-03-25T16:37:10.250 INFO:teuthology.orchestra.run.smithi167.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 173 kB/s | 58 kB 00:00 2024-03-25T16:37:10.275 INFO:teuthology.orchestra.run.smithi167.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 913 kB/s | 548 kB 00:00 2024-03-25T16:37:10.376 INFO:teuthology.orchestra.run.smithi167.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 407 kB/s | 51 kB 00:00 2024-03-25T16:37:10.409 INFO:teuthology.orchestra.run.smithi167.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 135 kB/s | 138 kB 00:01 2024-03-25T16:37:10.410 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:37:10.410 INFO:teuthology.orchestra.run.smithi167.stdout:Total 4.7 MB/s | 6.8 MB 00:01 2024-03-25T16:37:10.424 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-03-25T16:37:10.434 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-03-25T16:37:10.435 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-03-25T16:37:10.523 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-03-25T16:37:10.523 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-03-25T16:37:10.827 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-03-25T16:37:11.009 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T16:37:11.061 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T16:37:11.585 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T16:37:11.672 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T16:37:11.742 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T16:37:11.765 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:37:12.540 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:37:12.540 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T16:37:12.540 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T16:37:12.540 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T16:37:12.540 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T16:37:12.540 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T16:37:12.853 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T16:37:12.853 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:37:12.853 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-03-25T16:37:12.853 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:37:12.853 INFO:teuthology.orchestra.run.smithi167.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T16:37:12.853 INFO:teuthology.orchestra.run.smithi167.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T16:37:12.853 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T16:37:12.853 INFO:teuthology.orchestra.run.smithi167.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T16:37:12.853 INFO:teuthology.orchestra.run.smithi167.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T16:37:12.853 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:37:12.853 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-03-25T16:37:13.094 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-fuse 2024-03-25T16:37:13.667 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:06:38 ago on Mon 25 Mar 2024 04:30:35 PM UTC. 2024-03-25T16:37:13.818 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-03-25T16:37:13.818 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T16:37:13.818 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2024-03-25T16:37:13.818 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T16:37:13.819 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-03-25T16:37:13.819 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T16:37:13.819 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2024-03-25T16:37:13.819 INFO:teuthology.orchestra.run.smithi167.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T16:37:13.819 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:37:13.819 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-03-25T16:37:13.819 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T16:37:13.819 INFO:teuthology.orchestra.run.smithi167.stdout:Install 2 Packages 2024-03-25T16:37:13.819 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:37:13.820 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 964 k 2024-03-25T16:37:13.820 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 2.8 M 2024-03-25T16:37:13.820 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-03-25T16:37:13.953 INFO:teuthology.orchestra.run.smithi167.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 590 kB/s | 79 kB 00:00 2024-03-25T16:37:16.845 INFO:teuthology.orchestra.run.smithi167.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 293 kB/s | 885 kB 00:03 2024-03-25T16:37:16.846 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:37:16.847 INFO:teuthology.orchestra.run.smithi167.stdout:Total 318 kB/s | 964 kB 00:03 2024-03-25T16:37:16.847 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-03-25T16:37:16.870 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-03-25T16:37:16.870 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-03-25T16:37:16.947 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-03-25T16:37:16.947 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-03-25T16:37:17.141 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-03-25T16:37:17.288 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T16:37:17.353 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:37:18.002 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:37:18.002 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:37:18.295 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T16:37:18.295 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:37:18.295 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-03-25T16:37:18.295 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T16:37:18.295 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:37:18.295 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-03-25T16:37:18.553 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-volume 2024-03-25T16:37:19.127 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:06:44 ago on Mon 25 Mar 2024 04:30:35 PM UTC. 2024-03-25T16:37:19.282 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-03-25T16:37:19.282 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T16:37:19.282 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2024-03-25T16:37:19.282 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T16:37:19.282 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-03-25T16:37:19.282 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T16:37:19.282 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2024-03-25T16:37:19.283 INFO:teuthology.orchestra.run.smithi167.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T16:37:19.283 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:37:19.283 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-03-25T16:37:19.283 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T16:37:19.283 INFO:teuthology.orchestra.run.smithi167.stdout:Install 2 Packages 2024-03-25T16:37:19.283 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:37:19.283 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 332 k 2024-03-25T16:37:19.284 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 1.5 M 2024-03-25T16:37:19.284 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-03-25T16:37:19.418 INFO:teuthology.orchestra.run.smithi167.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 513 kB/s | 69 kB 00:00 2024-03-25T16:37:21.235 INFO:teuthology.orchestra.run.smithi167.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 135 kB/s | 264 kB 00:01 2024-03-25T16:37:21.236 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:37:21.237 INFO:teuthology.orchestra.run.smithi167.stdout:Total 170 kB/s | 332 kB 00:01 2024-03-25T16:37:21.237 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-03-25T16:37:21.247 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-03-25T16:37:21.247 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-03-25T16:37:21.290 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-03-25T16:37:21.291 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-03-25T16:37:21.442 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-03-25T16:37:21.651 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T16:37:21.688 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:37:21.703 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:37:21.703 INFO:teuthology.orchestra.run.smithi167.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:37:21.703 INFO:teuthology.orchestra.run.smithi167.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T16:37:21.703 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:37:22.410 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T16:37:22.730 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T16:37:22.731 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:37:22.731 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-03-25T16:37:22.731 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:37:22.731 INFO:teuthology.orchestra.run.smithi167.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T16:37:22.731 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:37:22.731 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-03-25T16:37:23.085 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install librados-devel 2024-03-25T16:37:23.671 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:06:48 ago on Mon 25 Mar 2024 04:30:35 PM UTC. 2024-03-25T16:37:23.820 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-03-25T16:37:23.821 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T16:37:23.821 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-03-25T16:37:23.821 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T16:37:23.822 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-03-25T16:37:23.822 INFO:teuthology.orchestra.run.smithi167.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T16:37:23.822 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:37:23.822 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-03-25T16:37:23.822 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T16:37:23.822 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-03-25T16:37:23.822 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:37:23.822 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 127 k 2024-03-25T16:37:23.822 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 456 k 2024-03-25T16:37:23.822 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-03-25T16:37:24.584 INFO:teuthology.orchestra.run.smithi167.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 166 kB/s | 127 kB 00:00 2024-03-25T16:37:24.584 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:37:24.584 INFO:teuthology.orchestra.run.smithi167.stdout:Total 166 kB/s | 127 kB 00:00 2024-03-25T16:37:24.585 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-03-25T16:37:24.591 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-03-25T16:37:24.591 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-03-25T16:37:24.648 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-03-25T16:37:24.649 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-03-25T16:37:24.812 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-03-25T16:37:24.915 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:37:25.372 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:37:25.647 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:37:25.647 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:37:25.647 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-03-25T16:37:25.647 INFO:teuthology.orchestra.run.smithi167.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:37:25.647 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:37:25.647 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-03-25T16:37:25.872 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install libcephfs2 2024-03-25T16:37:26.451 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:06:51 ago on Mon 25 Mar 2024 04:30:35 PM UTC. 2024-03-25T16:37:26.551 INFO:teuthology.orchestra.run.smithi167.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:37:26.607 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-03-25T16:37:26.608 INFO:teuthology.orchestra.run.smithi167.stdout:Nothing to do. 2024-03-25T16:37:26.608 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-03-25T16:37:26.674 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install libcephfs-devel 2024-03-25T16:37:27.245 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:06:52 ago on Mon 25 Mar 2024 04:30:35 PM UTC. 2024-03-25T16:37:27.394 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-03-25T16:37:27.395 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T16:37:27.395 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-03-25T16:37:27.395 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T16:37:27.395 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-03-25T16:37:27.395 INFO:teuthology.orchestra.run.smithi167.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T16:37:27.395 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:37:27.395 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-03-25T16:37:27.395 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T16:37:27.395 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-03-25T16:37:27.395 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:37:27.396 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 31 k 2024-03-25T16:37:27.396 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 140 k 2024-03-25T16:37:27.396 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-03-25T16:37:27.889 INFO:teuthology.orchestra.run.smithi167.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 63 kB/s | 31 kB 00:00 2024-03-25T16:37:27.890 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:37:27.890 INFO:teuthology.orchestra.run.smithi167.stdout:Total 63 kB/s | 31 kB 00:00 2024-03-25T16:37:27.890 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-03-25T16:37:27.893 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-03-25T16:37:27.893 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-03-25T16:37:27.921 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-03-25T16:37:27.921 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-03-25T16:37:28.039 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-03-25T16:37:28.144 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:37:28.487 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:37:28.752 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:37:28.752 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:37:28.752 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-03-25T16:37:28.752 INFO:teuthology.orchestra.run.smithi167.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:37:28.752 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:37:28.753 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-03-25T16:37:28.915 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install librados2 2024-03-25T16:37:29.497 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:06:54 ago on Mon 25 Mar 2024 04:30:35 PM UTC. 2024-03-25T16:37:29.600 INFO:teuthology.orchestra.run.smithi167.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:37:29.655 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-03-25T16:37:29.656 INFO:teuthology.orchestra.run.smithi167.stdout:Nothing to do. 2024-03-25T16:37:29.656 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-03-25T16:37:29.716 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install librbd1 2024-03-25T16:37:30.314 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:06:55 ago on Mon 25 Mar 2024 04:30:35 PM UTC. 2024-03-25T16:37:30.415 INFO:teuthology.orchestra.run.smithi167.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:37:30.468 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-03-25T16:37:30.469 INFO:teuthology.orchestra.run.smithi167.stdout:Nothing to do. 2024-03-25T16:37:30.469 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-03-25T16:37:30.525 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install python3-rados 2024-03-25T16:37:31.097 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:06:56 ago on Mon 25 Mar 2024 04:30:35 PM UTC. 2024-03-25T16:37:31.193 INFO:teuthology.orchestra.run.smithi167.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:37:31.245 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-03-25T16:37:31.246 INFO:teuthology.orchestra.run.smithi167.stdout:Nothing to do. 2024-03-25T16:37:31.246 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-03-25T16:37:31.296 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install python3-rgw 2024-03-25T16:37:31.860 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:06:56 ago on Mon 25 Mar 2024 04:30:35 PM UTC. 2024-03-25T16:37:31.956 INFO:teuthology.orchestra.run.smithi167.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:37:32.008 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-03-25T16:37:32.009 INFO:teuthology.orchestra.run.smithi167.stdout:Nothing to do. 2024-03-25T16:37:32.009 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-03-25T16:37:32.064 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install python3-cephfs 2024-03-25T16:37:32.630 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:06:57 ago on Mon 25 Mar 2024 04:30:35 PM UTC. 2024-03-25T16:37:32.726 INFO:teuthology.orchestra.run.smithi167.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:37:32.778 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-03-25T16:37:32.779 INFO:teuthology.orchestra.run.smithi167.stdout:Nothing to do. 2024-03-25T16:37:32.779 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-03-25T16:37:32.835 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install python3-rbd 2024-03-25T16:37:33.400 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:06:58 ago on Mon 25 Mar 2024 04:30:35 PM UTC. 2024-03-25T16:37:33.498 INFO:teuthology.orchestra.run.smithi167.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:37:33.550 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-03-25T16:37:33.551 INFO:teuthology.orchestra.run.smithi167.stdout:Nothing to do. 2024-03-25T16:37:33.551 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-03-25T16:37:33.601 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install rbd-fuse 2024-03-25T16:37:34.169 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:06:59 ago on Mon 25 Mar 2024 04:30:35 PM UTC. 2024-03-25T16:37:34.319 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-03-25T16:37:34.320 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T16:37:34.320 INFO:teuthology.orchestra.run.smithi167.stdout: Package Architecture Version Repository Size 2024-03-25T16:37:34.320 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T16:37:34.320 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-03-25T16:37:34.320 INFO:teuthology.orchestra.run.smithi167.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T16:37:34.320 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:37:34.320 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-03-25T16:37:34.320 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T16:37:34.320 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-03-25T16:37:34.321 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:37:34.321 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 85 k 2024-03-25T16:37:34.321 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 231 k 2024-03-25T16:37:34.321 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-03-25T16:37:35.334 INFO:teuthology.orchestra.run.smithi167.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 84 kB/s | 85 kB 00:01 2024-03-25T16:37:35.334 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:37:35.335 INFO:teuthology.orchestra.run.smithi167.stdout:Total 84 kB/s | 85 kB 00:01 2024-03-25T16:37:35.335 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-03-25T16:37:35.342 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-03-25T16:37:35.343 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-03-25T16:37:35.400 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-03-25T16:37:35.400 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-03-25T16:37:35.541 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-03-25T16:37:35.642 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:37:36.063 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:37:36.341 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:37:36.342 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:37:36.342 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-03-25T16:37:36.342 INFO:teuthology.orchestra.run.smithi167.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:37:36.342 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:37:36.342 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-03-25T16:37:36.586 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install rbd-mirror 2024-03-25T16:37:37.163 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:07:02 ago on Mon 25 Mar 2024 04:30:35 PM UTC. 2024-03-25T16:37:37.320 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-03-25T16:37:37.321 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T16:37:37.321 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-03-25T16:37:37.321 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T16:37:37.321 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-03-25T16:37:37.321 INFO:teuthology.orchestra.run.smithi167.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:37:37.321 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:37:37.321 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-03-25T16:37:37.321 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T16:37:37.321 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-03-25T16:37:37.321 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:37:37.321 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 3.1 M 2024-03-25T16:37:37.322 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 13 M 2024-03-25T16:37:37.322 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-03-25T16:37:49.196 INFO:teuthology.orchestra.run.smithi167.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 266 kB/s | 3.1 MB 00:11 2024-03-25T16:37:49.196 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:37:49.197 INFO:teuthology.orchestra.run.smithi167.stdout:Total 266 kB/s | 3.1 MB 00:11 2024-03-25T16:37:49.197 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-03-25T16:37:49.208 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-03-25T16:37:49.208 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-03-25T16:37:49.286 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-03-25T16:37:49.287 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-03-25T16:37:49.711 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-03-25T16:37:49.763 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:37:49.787 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:37:49.787 INFO:teuthology.orchestra.run.smithi167.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:37:49.787 INFO:teuthology.orchestra.run.smithi167.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T16:37:49.787 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:37:49.787 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:37:49.788 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:37:51.014 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:37:51.015 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:37:51.015 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-03-25T16:37:51.015 INFO:teuthology.orchestra.run.smithi167.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:37:51.015 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:37:51.015 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-03-25T16:37:51.232 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install rbd-nbd 2024-03-25T16:37:51.809 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:07:16 ago on Mon 25 Mar 2024 04:30:35 PM UTC. 2024-03-25T16:37:51.957 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-03-25T16:37:51.958 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T16:37:51.958 INFO:teuthology.orchestra.run.smithi167.stdout: Package Architecture Version Repository Size 2024-03-25T16:37:51.958 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T16:37:51.958 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-03-25T16:37:51.958 INFO:teuthology.orchestra.run.smithi167.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T16:37:51.958 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:37:51.958 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-03-25T16:37:51.959 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T16:37:51.959 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-03-25T16:37:51.959 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:37:51.959 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 171 k 2024-03-25T16:37:51.959 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 494 k 2024-03-25T16:37:51.959 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-03-25T16:37:53.242 INFO:teuthology.orchestra.run.smithi167.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 134 kB/s | 171 kB 00:01 2024-03-25T16:37:53.243 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:37:53.243 INFO:teuthology.orchestra.run.smithi167.stdout:Total 134 kB/s | 171 kB 00:01 2024-03-25T16:37:53.243 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-03-25T16:37:53.252 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-03-25T16:37:53.252 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-03-25T16:37:53.311 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-03-25T16:37:53.311 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-03-25T16:37:53.456 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-03-25T16:37:53.565 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:37:54.021 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:37:54.306 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:37:54.306 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:37:54.306 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-03-25T16:37:54.306 INFO:teuthology.orchestra.run.smithi167.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:37:54.306 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:37:54.306 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-03-25T16:37:54.526 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-base-debuginfo 2024-03-25T16:37:55.102 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:07:20 ago on Mon 25 Mar 2024 04:30:35 PM UTC. 2024-03-25T16:37:55.255 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-03-25T16:37:55.256 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T16:37:55.256 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-03-25T16:37:55.256 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T16:37:55.256 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-03-25T16:37:55.256 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T16:37:55.256 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2024-03-25T16:37:55.256 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T16:37:55.256 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:37:55.256 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-03-25T16:37:55.256 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T16:37:55.256 INFO:teuthology.orchestra.run.smithi167.stdout:Install 2 Packages 2024-03-25T16:37:55.257 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:37:55.257 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 91 M 2024-03-25T16:37:55.257 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 498 M 2024-03-25T16:37:55.257 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-03-25T16:38:12.153 INFO:teuthology.orchestra.run.smithi155.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 273 kB/s | 22 MB 01:24 2024-03-25T16:38:56.474 INFO:teuthology.orchestra.run.smithi167.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 375 kB/s | 22 MB 01:01 2024-03-25T16:39:40.401 INFO:teuthology.orchestra.run.smithi155.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 405 kB/s | 68 MB 02:52 2024-03-25T16:39:40.402 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:39:40.402 INFO:teuthology.orchestra.run.smithi155.stdout:Total 538 kB/s | 91 MB 02:52 2024-03-25T16:39:40.402 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-25T16:39:40.406 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-25T16:39:40.406 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-25T16:39:40.996 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-25T16:39:40.996 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-25T16:39:44.143 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-25T16:39:51.224 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:39:51.301 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:39:54.661 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:39:54.661 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T16:39:54.927 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:39:54.928 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:39:54.928 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-25T16:39:54.928 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:39:54.928 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:39:54.928 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:39:54.928 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-25T16:39:55.169 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-common-debuginfo 2024-03-25T16:39:55.733 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:09:43 ago on Mon 25 Mar 2024 04:30:12 PM UTC. 2024-03-25T16:39:55.881 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-25T16:39:55.882 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T16:39:55.882 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-03-25T16:39:55.882 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T16:39:55.883 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-25T16:39:55.883 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T16:39:55.883 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:39:55.883 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-25T16:39:55.883 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T16:39:55.883 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-03-25T16:39:55.883 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:39:55.883 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 416 M 2024-03-25T16:39:55.883 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 1.7 G 2024-03-25T16:39:55.883 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-25T16:41:26.799 INFO:teuthology.orchestra.run.smithi167.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 330 kB/s | 68 MB 03:31 2024-03-25T16:41:26.800 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:41:26.800 INFO:teuthology.orchestra.run.smithi167.stdout:Total 438 kB/s | 91 MB 03:31 2024-03-25T16:41:26.800 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-03-25T16:41:26.804 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-03-25T16:41:26.804 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-03-25T16:41:27.399 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-03-25T16:41:27.400 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-03-25T16:41:30.496 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-03-25T16:41:37.682 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:41:37.850 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:41:41.224 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:41:41.224 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T16:41:41.489 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:41:41.490 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:41:41.490 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-03-25T16:41:41.490 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:41:41.490 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:41:41.490 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:41:41.490 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-03-25T16:41:41.699 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-common-debuginfo 2024-03-25T16:41:42.263 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:11:07 ago on Mon 25 Mar 2024 04:30:35 PM UTC. 2024-03-25T16:41:42.413 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-03-25T16:41:42.414 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T16:41:42.414 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-03-25T16:41:42.414 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T16:41:42.414 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-03-25T16:41:42.414 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T16:41:42.414 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:41:42.414 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-03-25T16:41:42.414 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T16:41:42.414 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-03-25T16:41:42.415 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T16:41:42.415 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 416 M 2024-03-25T16:41:42.415 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 1.7 G 2024-03-25T16:41:42.415 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-03-25T16:57:44.433 INFO:teuthology.orchestra.run.smithi155.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 399 kB/s | 416 MB 17:48 2024-03-25T16:57:44.434 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:57:44.434 INFO:teuthology.orchestra.run.smithi155.stdout:Total 399 kB/s | 416 MB 17:48 2024-03-25T16:57:44.434 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-25T16:57:44.437 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-25T16:57:44.437 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-25T16:57:46.907 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-25T16:57:46.907 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-25T16:58:29.218 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-25T16:58:29.310 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:58:38.355 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:58:38.612 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:58:38.612 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:58:38.612 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-25T16:58:38.612 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:58:38.612 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:58:38.612 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-25T16:58:38.893 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T16:58:39.457 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:28:27 ago on Mon 25 Mar 2024 04:30:12 PM UTC. 2024-03-25T16:58:39.605 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-25T16:58:39.606 INFO:teuthology.orchestra.run.smithi155.stdout:======================================================================================== 2024-03-25T16:58:39.606 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-03-25T16:58:39.606 INFO:teuthology.orchestra.run.smithi155.stdout:======================================================================================== 2024-03-25T16:58:39.606 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-25T16:58:39.606 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T16:58:39.606 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:58:39.606 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-25T16:58:39.607 INFO:teuthology.orchestra.run.smithi155.stdout:======================================================================================== 2024-03-25T16:58:39.607 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-03-25T16:58:39.607 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:58:39.607 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 1.4 M 2024-03-25T16:58:39.607 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 6.4 M 2024-03-25T16:58:39.607 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-25T16:58:43.197 INFO:teuthology.orchestra.run.smithi155.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 407 kB/s | 1.4 MB 00:03 2024-03-25T16:58:43.198 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:58:43.198 INFO:teuthology.orchestra.run.smithi155.stdout:Total 407 kB/s | 1.4 MB 00:03 2024-03-25T16:58:43.199 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-25T16:58:43.202 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-25T16:58:43.202 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-25T16:58:43.247 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-25T16:58:43.247 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-25T16:58:43.504 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-25T16:58:43.613 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:58:44.058 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:58:44.326 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:58:44.326 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:58:44.326 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-25T16:58:44.326 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:58:44.326 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:58:44.326 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-25T16:58:44.506 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T16:58:45.075 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:28:33 ago on Mon 25 Mar 2024 04:30:12 PM UTC. 2024-03-25T16:58:45.225 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-25T16:58:45.226 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T16:58:45.226 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-03-25T16:58:45.226 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T16:58:45.226 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-25T16:58:45.226 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T16:58:45.226 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:58:45.227 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-25T16:58:45.227 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T16:58:45.227 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-03-25T16:58:45.227 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T16:58:45.227 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 176 M 2024-03-25T16:58:45.227 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 753 M 2024-03-25T16:58:45.227 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-25T17:06:49.929 INFO:teuthology.orchestra.run.smithi167.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 283 kB/s | 416 MB 25:07 2024-03-25T17:06:49.929 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:06:49.929 INFO:teuthology.orchestra.run.smithi167.stdout:Total 283 kB/s | 416 MB 25:07 2024-03-25T17:06:49.929 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-03-25T17:06:49.932 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-03-25T17:06:49.932 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-03-25T17:06:52.403 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-03-25T17:06:52.403 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-03-25T17:07:34.794 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-03-25T17:07:34.898 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:07:43.164 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:07:43.428 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:07:43.428 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T17:07:43.429 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-03-25T17:07:43.429 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:07:43.429 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T17:07:43.429 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-03-25T17:07:43.670 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T17:07:44.235 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:37:09 ago on Mon 25 Mar 2024 04:30:35 PM UTC. 2024-03-25T17:07:44.383 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-03-25T17:07:44.383 INFO:teuthology.orchestra.run.smithi167.stdout:======================================================================================== 2024-03-25T17:07:44.384 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-03-25T17:07:44.384 INFO:teuthology.orchestra.run.smithi167.stdout:======================================================================================== 2024-03-25T17:07:44.384 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-03-25T17:07:44.384 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T17:07:44.384 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T17:07:44.384 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-03-25T17:07:44.384 INFO:teuthology.orchestra.run.smithi167.stdout:======================================================================================== 2024-03-25T17:07:44.384 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-03-25T17:07:44.384 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T17:07:44.385 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 1.4 M 2024-03-25T17:07:44.385 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 6.4 M 2024-03-25T17:07:44.385 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-03-25T17:07:49.343 INFO:teuthology.orchestra.run.smithi167.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 295 kB/s | 1.4 MB 00:04 2024-03-25T17:07:49.344 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:07:49.344 INFO:teuthology.orchestra.run.smithi167.stdout:Total 295 kB/s | 1.4 MB 00:04 2024-03-25T17:07:49.344 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-03-25T17:07:49.348 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-03-25T17:07:49.348 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-03-25T17:07:49.392 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-03-25T17:07:49.393 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-03-25T17:07:49.647 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-03-25T17:07:49.758 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:07:50.221 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:07:50.505 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:07:50.505 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T17:07:50.505 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-03-25T17:07:50.505 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:07:50.505 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T17:07:50.505 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-03-25T17:07:50.682 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T17:07:51.248 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:37:16 ago on Mon 25 Mar 2024 04:30:35 PM UTC. 2024-03-25T17:07:51.396 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-03-25T17:07:51.397 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T17:07:51.397 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-03-25T17:07:51.397 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T17:07:51.397 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-03-25T17:07:51.397 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T17:07:51.397 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T17:07:51.397 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-03-25T17:07:51.397 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T17:07:51.397 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-03-25T17:07:51.397 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T17:07:51.397 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 176 M 2024-03-25T17:07:51.398 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 753 M 2024-03-25T17:07:51.398 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-03-25T17:08:10.827 INFO:teuthology.orchestra.run.smithi155.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 318 kB/s | 176 MB 09:25 2024-03-25T17:08:10.828 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:08:10.828 INFO:teuthology.orchestra.run.smithi155.stdout:Total 318 kB/s | 176 MB 09:25 2024-03-25T17:08:10.828 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-25T17:08:10.831 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-25T17:08:10.831 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-25T17:08:11.889 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-25T17:08:11.890 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-25T17:08:30.161 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-25T17:08:30.330 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:08:34.973 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:08:35.242 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:08:35.243 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:08:35.243 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-25T17:08:35.243 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:08:35.243 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:08:35.243 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-25T17:08:35.475 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-test-debuginfo 2024-03-25T17:08:36.042 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:38:24 ago on Mon 25 Mar 2024 04:30:12 PM UTC. 2024-03-25T17:08:36.192 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-25T17:08:36.192 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T17:08:36.193 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-03-25T17:08:36.193 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T17:08:36.193 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-25T17:08:36.193 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T17:08:36.193 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:08:36.193 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-25T17:08:36.193 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T17:08:36.193 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-03-25T17:08:36.193 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:08:36.193 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 672 M 2024-03-25T17:08:36.193 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 3.1 G 2024-03-25T17:08:36.193 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-25T17:18:18.454 INFO:teuthology.orchestra.run.smithi167.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 287 kB/s | 176 MB 10:27 2024-03-25T17:18:18.479 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:18:18.479 INFO:teuthology.orchestra.run.smithi167.stdout:Total 287 kB/s | 176 MB 10:27 2024-03-25T17:18:18.480 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-03-25T17:18:18.480 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-03-25T17:18:18.480 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-03-25T17:18:19.519 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-03-25T17:18:19.519 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-03-25T17:18:37.554 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-03-25T17:18:37.683 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:18:42.419 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:18:42.684 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:18:42.684 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T17:18:42.684 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-03-25T17:18:42.684 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:18:42.684 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T17:18:42.684 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-03-25T17:18:42.884 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-test-debuginfo 2024-03-25T17:18:43.450 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:48:08 ago on Mon 25 Mar 2024 04:30:35 PM UTC. 2024-03-25T17:18:43.600 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-03-25T17:18:43.601 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T17:18:43.601 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-03-25T17:18:43.601 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T17:18:43.601 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-03-25T17:18:43.601 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T17:18:43.601 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T17:18:43.601 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-03-25T17:18:43.602 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T17:18:43.602 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-03-25T17:18:43.602 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T17:18:43.602 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 672 M 2024-03-25T17:18:43.602 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 3.1 G 2024-03-25T17:18:43.602 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-03-25T17:35:30.191 INFO:teuthology.orchestra.run.smithi155.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 426 kB/s | 672 MB 26:53 2024-03-25T17:35:30.209 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:35:30.209 INFO:teuthology.orchestra.run.smithi155.stdout:Total 426 kB/s | 672 MB 26:53 2024-03-25T17:35:30.209 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-25T17:35:30.210 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-25T17:35:30.210 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-25T17:35:34.154 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-25T17:35:34.154 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-25T17:36:45.042 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-25T17:36:45.182 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:36:54.097 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:36:54.387 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:36:54.387 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:36:54.387 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-25T17:36:54.387 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:36:54.387 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:36:54.387 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-25T17:36:54.666 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-base-debuginfo 2024-03-25T17:36:55.237 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 1:06:43 ago on Mon 25 Mar 2024 04:30:12 PM UTC. 2024-03-25T17:36:55.334 INFO:teuthology.orchestra.run.smithi155.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:36:55.386 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-25T17:36:55.387 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-03-25T17:36:55.387 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-25T17:36:55.446 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T17:36:56.012 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 1:06:44 ago on Mon 25 Mar 2024 04:30:12 PM UTC. 2024-03-25T17:36:56.162 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-25T17:36:56.162 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T17:36:56.163 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-03-25T17:36:56.163 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T17:36:56.163 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-25T17:36:56.163 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T17:36:56.163 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:36:56.163 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-25T17:36:56.163 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T17:36:56.163 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-03-25T17:36:56.163 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:36:56.163 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 20 M 2024-03-25T17:36:56.163 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 103 M 2024-03-25T17:36:56.163 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-25T17:38:03.413 INFO:teuthology.orchestra.run.smithi155.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 300 kB/s | 20 MB 01:07 2024-03-25T17:38:03.413 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:38:03.413 INFO:teuthology.orchestra.run.smithi155.stdout:Total 300 kB/s | 20 MB 01:07 2024-03-25T17:38:03.414 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-25T17:38:03.417 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-25T17:38:03.417 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-25T17:38:03.568 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-25T17:38:03.568 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-25T17:38:05.798 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-25T17:38:05.976 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:38:06.908 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:38:07.173 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:38:07.173 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:38:07.173 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-25T17:38:07.173 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:38:07.173 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:38:07.173 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-25T17:38:07.339 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T17:38:07.908 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 1:07:55 ago on Mon 25 Mar 2024 04:30:12 PM UTC. 2024-03-25T17:38:08.056 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-25T17:38:08.057 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T17:38:08.057 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-03-25T17:38:08.057 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T17:38:08.057 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-25T17:38:08.057 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T17:38:08.057 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:38:08.057 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-25T17:38:08.057 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T17:38:08.058 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-03-25T17:38:08.058 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:38:08.058 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 48 M 2024-03-25T17:38:08.058 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 173 M 2024-03-25T17:38:08.058 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-25T17:40:18.099 INFO:teuthology.orchestra.run.smithi155.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 376 kB/s | 48 MB 02:10 2024-03-25T17:40:18.100 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:40:18.100 INFO:teuthology.orchestra.run.smithi155.stdout:Total 376 kB/s | 48 MB 02:10 2024-03-25T17:40:18.100 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-25T17:40:18.102 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-25T17:40:18.103 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-25T17:40:18.415 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-25T17:40:18.415 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-25T17:40:23.175 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-25T17:40:23.303 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:40:24.706 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:40:24.963 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:40:24.963 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:40:24.963 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-25T17:40:24.963 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:40:24.963 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:40:24.963 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-25T17:40:25.139 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T17:40:25.707 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 1:10:13 ago on Mon 25 Mar 2024 04:30:12 PM UTC. 2024-03-25T17:40:25.858 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-25T17:40:25.859 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T17:40:25.859 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-03-25T17:40:25.859 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T17:40:25.859 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-25T17:40:25.859 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T17:40:25.859 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:40:25.860 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-25T17:40:25.860 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T17:40:25.860 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-03-25T17:40:25.860 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:40:25.860 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 99 M 2024-03-25T17:40:25.860 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 449 M 2024-03-25T17:40:25.860 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-25T17:43:35.751 INFO:teuthology.orchestra.run.smithi155.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 531 kB/s | 99 MB 03:09 2024-03-25T17:43:35.752 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:43:35.752 INFO:teuthology.orchestra.run.smithi155.stdout:Total 531 kB/s | 99 MB 03:09 2024-03-25T17:43:35.752 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-25T17:43:35.754 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-25T17:43:35.755 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-25T17:43:36.362 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-25T17:43:36.363 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-25T17:43:46.615 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-25T17:43:46.730 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:43:49.590 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:43:49.854 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:43:49.855 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:43:49.855 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-25T17:43:49.855 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:43:49.855 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:43:49.855 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-25T17:43:50.039 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T17:43:50.606 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 1:13:38 ago on Mon 25 Mar 2024 04:30:12 PM UTC. 2024-03-25T17:43:50.756 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-25T17:43:50.757 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T17:43:50.757 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-03-25T17:43:50.757 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T17:43:50.757 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-25T17:43:50.757 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T17:43:50.757 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:43:50.757 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-25T17:43:50.757 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T17:43:50.757 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-03-25T17:43:50.757 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:43:50.757 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 287 M 2024-03-25T17:43:50.758 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 1.2 G 2024-03-25T17:43:50.758 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-25T17:52:55.700 INFO:teuthology.orchestra.run.smithi155.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 539 kB/s | 287 MB 09:04 2024-03-25T17:52:55.701 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:52:55.701 INFO:teuthology.orchestra.run.smithi155.stdout:Total 539 kB/s | 287 MB 09:04 2024-03-25T17:52:55.701 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-25T17:52:55.703 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-25T17:52:55.703 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-25T17:52:57.406 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-25T17:52:57.406 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-25T17:53:26.814 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-25T17:53:27.112 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:53:28.313 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:53:28.590 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:53:28.590 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:53:28.590 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-25T17:53:28.590 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:53:28.590 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:53:28.590 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-25T17:53:28.801 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T17:53:29.365 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 1:23:17 ago on Mon 25 Mar 2024 04:30:12 PM UTC. 2024-03-25T17:53:29.514 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-25T17:53:29.515 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T17:53:29.515 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-03-25T17:53:29.515 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T17:53:29.515 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-25T17:53:29.516 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T17:53:29.516 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:53:29.516 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-25T17:53:29.516 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T17:53:29.516 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-03-25T17:53:29.516 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:53:29.516 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 5.9 M 2024-03-25T17:53:29.516 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 28 M 2024-03-25T17:53:29.516 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-25T17:53:38.730 INFO:teuthology.orchestra.run.smithi167.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 329 kB/s | 672 MB 34:55 2024-03-25T17:53:38.731 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:53:38.731 INFO:teuthology.orchestra.run.smithi167.stdout:Total 329 kB/s | 672 MB 34:55 2024-03-25T17:53:38.731 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-03-25T17:53:38.737 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-03-25T17:53:38.737 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-03-25T17:53:42.703 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-03-25T17:53:42.703 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-03-25T17:53:46.347 INFO:teuthology.orchestra.run.smithi155.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 362 kB/s | 5.9 MB 00:16 2024-03-25T17:53:46.348 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:53:46.348 INFO:teuthology.orchestra.run.smithi155.stdout:Total 362 kB/s | 5.9 MB 00:16 2024-03-25T17:53:46.349 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-25T17:53:46.353 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-25T17:53:46.353 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-25T17:53:46.426 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-25T17:53:46.427 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-25T17:53:47.165 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-25T17:53:47.288 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:53:47.891 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:53:48.164 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:53:48.164 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:53:48.164 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-25T17:53:48.164 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:53:48.164 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:53:48.164 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-25T17:53:48.357 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install librados-devel-debuginfo 2024-03-25T17:53:48.922 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 1:23:36 ago on Mon 25 Mar 2024 04:30:12 PM UTC. 2024-03-25T17:53:49.072 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-25T17:53:49.073 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T17:53:49.073 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-03-25T17:53:49.073 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T17:53:49.073 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-25T17:53:49.073 INFO:teuthology.orchestra.run.smithi155.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T17:53:49.073 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:53:49.073 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-25T17:53:49.074 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T17:53:49.074 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-03-25T17:53:49.074 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:53:49.074 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 109 k 2024-03-25T17:53:49.074 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 440 k 2024-03-25T17:53:49.074 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-25T17:53:49.759 INFO:teuthology.orchestra.run.smithi155.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 159 kB/s | 109 kB 00:00 2024-03-25T17:53:49.759 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:53:49.760 INFO:teuthology.orchestra.run.smithi155.stdout:Total 158 kB/s | 109 kB 00:00 2024-03-25T17:53:49.760 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-25T17:53:49.763 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-25T17:53:49.764 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-25T17:53:49.799 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-25T17:53:49.799 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-25T17:53:49.918 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-25T17:53:50.016 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:53:50.418 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:53:50.684 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:53:50.684 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:53:50.684 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-25T17:53:50.684 INFO:teuthology.orchestra.run.smithi155.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:53:50.684 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:53:50.684 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-25T17:53:50.881 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T17:53:51.448 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 1:23:39 ago on Mon 25 Mar 2024 04:30:12 PM UTC. 2024-03-25T17:53:51.597 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-25T17:53:51.597 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T17:53:51.597 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-03-25T17:53:51.597 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T17:53:51.597 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-25T17:53:51.598 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T17:53:51.598 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:53:51.598 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-25T17:53:51.598 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T17:53:51.598 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-03-25T17:53:51.598 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:53:51.598 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 5.4 M 2024-03-25T17:53:51.599 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 25 M 2024-03-25T17:53:51.599 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-25T17:54:06.369 INFO:teuthology.orchestra.run.smithi155.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 371 kB/s | 5.4 MB 00:14 2024-03-25T17:54:06.370 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:54:06.370 INFO:teuthology.orchestra.run.smithi155.stdout:Total 371 kB/s | 5.4 MB 00:14 2024-03-25T17:54:06.370 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-25T17:54:06.374 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-25T17:54:06.374 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-25T17:54:06.443 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-25T17:54:06.444 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-25T17:54:07.111 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-25T17:54:07.217 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:54:07.732 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:54:08.000 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:54:08.000 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:54:08.000 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-25T17:54:08.000 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:54:08.001 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:54:08.001 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-25T17:54:08.208 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install librados2-debuginfo 2024-03-25T17:54:08.774 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 1:23:56 ago on Mon 25 Mar 2024 04:30:12 PM UTC. 2024-03-25T17:54:08.924 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-25T17:54:08.924 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T17:54:08.924 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-03-25T17:54:08.924 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T17:54:08.924 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-25T17:54:08.924 INFO:teuthology.orchestra.run.smithi155.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T17:54:08.925 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:54:08.925 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-25T17:54:08.925 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T17:54:08.925 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-03-25T17:54:08.925 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:54:08.926 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 44 M 2024-03-25T17:54:08.926 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 253 M 2024-03-25T17:54:08.926 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-25T17:54:54.002 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-03-25T17:54:54.215 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:55:03.960 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:55:04.217 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:55:04.217 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T17:55:04.217 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-03-25T17:55:04.217 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:55:04.217 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T17:55:04.217 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-03-25T17:55:04.509 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-base-debuginfo 2024-03-25T17:55:05.082 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 1:24:30 ago on Mon 25 Mar 2024 04:30:35 PM UTC. 2024-03-25T17:55:05.179 INFO:teuthology.orchestra.run.smithi167.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:55:05.231 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-03-25T17:55:05.232 INFO:teuthology.orchestra.run.smithi167.stdout:Nothing to do. 2024-03-25T17:55:05.232 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-03-25T17:55:05.307 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T17:55:05.873 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 1:24:30 ago on Mon 25 Mar 2024 04:30:35 PM UTC. 2024-03-25T17:55:06.022 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-03-25T17:55:06.023 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T17:55:06.023 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-03-25T17:55:06.023 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T17:55:06.023 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-03-25T17:55:06.023 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T17:55:06.023 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T17:55:06.023 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-03-25T17:55:06.023 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T17:55:06.023 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-03-25T17:55:06.023 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T17:55:06.023 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 20 M 2024-03-25T17:55:06.024 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 103 M 2024-03-25T17:55:06.024 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-03-25T17:55:58.730 INFO:teuthology.orchestra.run.smithi167.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 383 kB/s | 20 MB 00:52 2024-03-25T17:55:58.731 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:55:58.731 INFO:teuthology.orchestra.run.smithi167.stdout:Total 383 kB/s | 20 MB 00:52 2024-03-25T17:55:58.731 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-03-25T17:55:58.734 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-03-25T17:55:58.734 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-03-25T17:55:58.883 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-03-25T17:55:58.884 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-03-25T17:56:01.118 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-03-25T17:56:01.227 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:56:02.237 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:56:02.492 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:56:02.492 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T17:56:02.492 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-03-25T17:56:02.493 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:56:02.493 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T17:56:02.493 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-03-25T17:56:02.668 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T17:56:03.236 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 1:25:28 ago on Mon 25 Mar 2024 04:30:35 PM UTC. 2024-03-25T17:56:03.385 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-03-25T17:56:03.385 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T17:56:03.385 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-03-25T17:56:03.385 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T17:56:03.386 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-03-25T17:56:03.386 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T17:56:03.386 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T17:56:03.386 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-03-25T17:56:03.386 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T17:56:03.386 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-03-25T17:56:03.386 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T17:56:03.386 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 48 M 2024-03-25T17:56:03.387 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 173 M 2024-03-25T17:56:03.387 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-03-25T17:56:17.939 INFO:teuthology.orchestra.run.smithi155.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 351 kB/s | 44 MB 02:09 2024-03-25T17:56:17.939 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:56:17.940 INFO:teuthology.orchestra.run.smithi155.stdout:Total 351 kB/s | 44 MB 02:09 2024-03-25T17:56:17.940 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-25T17:56:17.943 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-25T17:56:17.943 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-25T17:56:18.241 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-25T17:56:18.241 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-25T17:56:23.307 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-25T17:56:23.412 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:56:25.296 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:56:25.559 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:56:25.559 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:56:25.559 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-25T17:56:25.559 INFO:teuthology.orchestra.run.smithi155.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:56:25.560 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:56:25.560 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-25T17:56:25.753 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install librbd1-debuginfo 2024-03-25T17:56:26.312 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 1:26:14 ago on Mon 25 Mar 2024 04:30:12 PM UTC. 2024-03-25T17:56:26.461 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-25T17:56:26.463 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T17:56:26.463 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-03-25T17:56:26.463 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T17:56:26.463 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-25T17:56:26.463 INFO:teuthology.orchestra.run.smithi155.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T17:56:26.463 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:56:26.463 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-25T17:56:26.463 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T17:56:26.463 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-03-25T17:56:26.463 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:56:26.463 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 52 M 2024-03-25T17:56:26.463 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 308 M 2024-03-25T17:56:26.464 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-25T17:57:58.548 INFO:teuthology.orchestra.run.smithi155.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 576 kB/s | 52 MB 01:32 2024-03-25T17:57:58.548 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:57:58.548 INFO:teuthology.orchestra.run.smithi155.stdout:Total 576 kB/s | 52 MB 01:32 2024-03-25T17:57:58.548 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-25T17:57:58.551 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-25T17:57:58.551 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-25T17:57:58.896 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-25T17:57:58.896 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-25T17:58:04.886 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-25T17:58:05.020 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:58:07.189 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:58:07.465 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:58:07.465 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:58:07.465 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-25T17:58:07.465 INFO:teuthology.orchestra.run.smithi155.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:58:07.465 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:58:07.465 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-25T17:58:07.661 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T17:58:08.227 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 1:27:56 ago on Mon 25 Mar 2024 04:30:12 PM UTC. 2024-03-25T17:58:08.377 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-25T17:58:08.378 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T17:58:08.378 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-03-25T17:58:08.378 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T17:58:08.378 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-25T17:58:08.379 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T17:58:08.379 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:58:08.379 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-25T17:58:08.379 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T17:58:08.379 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-03-25T17:58:08.379 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:58:08.379 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 615 k 2024-03-25T17:58:08.379 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 2.1 M 2024-03-25T17:58:08.379 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-25T17:58:10.674 INFO:teuthology.orchestra.run.smithi155.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 268 kB/s | 615 kB 00:02 2024-03-25T17:58:10.674 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:58:10.675 INFO:teuthology.orchestra.run.smithi155.stdout:Total 268 kB/s | 615 kB 00:02 2024-03-25T17:58:10.675 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-25T17:58:10.679 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-25T17:58:10.679 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-25T17:58:10.720 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-25T17:58:10.721 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-25T17:58:10.889 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-25T17:58:10.997 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:58:11.410 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:58:11.687 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:58:11.687 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:58:11.687 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-25T17:58:11.687 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:58:11.688 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:58:11.688 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-25T17:58:11.883 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install python3-rados-debuginfo 2024-03-25T17:58:12.444 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 1:28:00 ago on Mon 25 Mar 2024 04:30:12 PM UTC. 2024-03-25T17:58:12.594 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-25T17:58:12.595 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T17:58:12.595 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-03-25T17:58:12.596 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T17:58:12.596 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-25T17:58:12.596 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T17:58:12.596 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:58:12.596 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-25T17:58:12.596 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T17:58:12.596 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-03-25T17:58:12.596 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:58:12.596 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 771 k 2024-03-25T17:58:12.596 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 2.7 M 2024-03-25T17:58:12.596 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-25T17:58:14.444 INFO:teuthology.orchestra.run.smithi155.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 417 kB/s | 771 kB 00:01 2024-03-25T17:58:14.444 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:58:14.445 INFO:teuthology.orchestra.run.smithi155.stdout:Total 417 kB/s | 771 kB 00:01 2024-03-25T17:58:14.445 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-25T17:58:14.449 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-25T17:58:14.449 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-25T17:58:14.489 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-25T17:58:14.490 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-25T17:58:14.669 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-25T17:58:14.775 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T17:58:15.191 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T17:58:15.460 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T17:58:15.460 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:58:15.460 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-25T17:58:15.460 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:58:15.460 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:58:15.460 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-25T17:58:15.674 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T17:58:16.236 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 1:28:04 ago on Mon 25 Mar 2024 04:30:12 PM UTC. 2024-03-25T17:58:16.385 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-25T17:58:16.386 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T17:58:16.386 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-03-25T17:58:16.386 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T17:58:16.386 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-25T17:58:16.386 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T17:58:16.386 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:58:16.386 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-25T17:58:16.386 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T17:58:16.386 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-03-25T17:58:16.387 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:58:16.387 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 1.1 M 2024-03-25T17:58:16.387 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 4.3 M 2024-03-25T17:58:16.387 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-25T17:58:19.546 INFO:teuthology.orchestra.run.smithi155.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 364 kB/s | 1.1 MB 00:03 2024-03-25T17:58:19.547 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:58:19.547 INFO:teuthology.orchestra.run.smithi155.stdout:Total 364 kB/s | 1.1 MB 00:03 2024-03-25T17:58:19.548 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-25T17:58:19.552 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-25T17:58:19.552 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-25T17:58:19.594 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-25T17:58:19.595 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-25T17:58:19.816 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-25T17:58:19.914 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:58:20.292 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:58:20.559 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:58:20.559 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:58:20.559 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-25T17:58:20.559 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:58:20.560 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:58:20.560 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-25T17:58:20.783 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T17:58:21.348 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 1:28:09 ago on Mon 25 Mar 2024 04:30:12 PM UTC. 2024-03-25T17:58:21.499 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-25T17:58:21.499 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T17:58:21.500 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-03-25T17:58:21.500 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T17:58:21.500 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-25T17:58:21.500 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T17:58:21.500 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:58:21.500 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-25T17:58:21.500 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T17:58:21.500 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-03-25T17:58:21.500 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:58:21.500 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 337 k 2024-03-25T17:58:21.500 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 1.0 M 2024-03-25T17:58:21.500 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-25T17:58:22.877 INFO:teuthology.orchestra.run.smithi155.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 245 kB/s | 337 kB 00:01 2024-03-25T17:58:22.877 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:58:22.878 INFO:teuthology.orchestra.run.smithi155.stdout:Total 245 kB/s | 337 kB 00:01 2024-03-25T17:58:22.878 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-25T17:58:22.882 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-25T17:58:22.882 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-25T17:58:22.922 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-25T17:58:22.923 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-25T17:58:23.058 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-25T17:58:23.159 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:58:23.521 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:58:23.793 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:58:23.793 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:58:23.793 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-25T17:58:23.793 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:58:23.793 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:58:23.793 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-25T17:58:24.003 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T17:58:24.561 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 1:28:12 ago on Mon 25 Mar 2024 04:30:12 PM UTC. 2024-03-25T17:58:24.710 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-25T17:58:24.710 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T17:58:24.710 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-03-25T17:58:24.711 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T17:58:24.711 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-25T17:58:24.711 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T17:58:24.711 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:58:24.711 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-25T17:58:24.711 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T17:58:24.711 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-03-25T17:58:24.711 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:58:24.711 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 509 k 2024-03-25T17:58:24.711 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 2.5 M 2024-03-25T17:58:24.711 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-25T17:58:26.280 INFO:teuthology.orchestra.run.smithi155.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 325 kB/s | 509 kB 00:01 2024-03-25T17:58:26.280 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:58:26.280 INFO:teuthology.orchestra.run.smithi155.stdout:Total 324 kB/s | 509 kB 00:01 2024-03-25T17:58:26.281 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-25T17:58:26.285 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-25T17:58:26.285 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-25T17:58:26.324 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-25T17:58:26.325 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-25T17:58:26.489 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-25T17:58:26.603 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:58:26.980 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:58:27.258 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:58:27.258 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:58:27.258 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-25T17:58:27.258 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:58:27.258 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:58:27.258 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-25T17:58:27.468 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T17:58:28.030 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 1:28:16 ago on Mon 25 Mar 2024 04:30:12 PM UTC. 2024-03-25T17:58:28.182 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-25T17:58:28.183 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T17:58:28.183 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-03-25T17:58:28.183 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T17:58:28.183 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-25T17:58:28.183 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T17:58:28.183 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:58:28.183 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-25T17:58:28.183 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T17:58:28.183 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-03-25T17:58:28.183 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T17:58:28.184 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 52 M 2024-03-25T17:58:28.184 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 298 M 2024-03-25T17:58:28.184 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-25T17:58:43.329 INFO:teuthology.orchestra.run.smithi167.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 306 kB/s | 48 MB 02:39 2024-03-25T17:58:43.330 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:58:43.330 INFO:teuthology.orchestra.run.smithi167.stdout:Total 306 kB/s | 48 MB 02:39 2024-03-25T17:58:43.330 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-03-25T17:58:43.333 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-03-25T17:58:43.333 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-03-25T17:58:43.645 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-03-25T17:58:43.645 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-03-25T17:58:48.386 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-03-25T17:58:48.509 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:58:49.934 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:58:50.192 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:58:50.193 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T17:58:50.193 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-03-25T17:58:50.193 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:58:50.193 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T17:58:50.193 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-03-25T17:58:50.375 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T17:58:50.941 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 1:28:15 ago on Mon 25 Mar 2024 04:30:35 PM UTC. 2024-03-25T17:58:51.090 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-03-25T17:58:51.090 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T17:58:51.091 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-03-25T17:58:51.091 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T17:58:51.091 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-03-25T17:58:51.091 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T17:58:51.091 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T17:58:51.091 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-03-25T17:58:51.091 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T17:58:51.091 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-03-25T17:58:51.091 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T17:58:51.091 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 99 M 2024-03-25T17:58:51.091 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 449 M 2024-03-25T17:58:51.091 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-03-25T18:00:26.052 INFO:teuthology.orchestra.run.smithi155.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 448 kB/s | 52 MB 01:57 2024-03-25T18:00:26.052 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:00:26.052 INFO:teuthology.orchestra.run.smithi155.stdout:Total 448 kB/s | 52 MB 01:57 2024-03-25T18:00:26.052 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-25T18:00:26.055 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-25T18:00:26.055 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-25T18:00:26.392 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-25T18:00:26.392 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-25T18:00:32.332 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-25T18:00:32.424 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:00:34.472 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:00:34.753 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:00:34.753 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T18:00:34.754 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-25T18:00:34.754 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:00:34.754 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T18:00:34.754 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-25T18:00:34.949 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T18:00:35.523 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 1:30:23 ago on Mon 25 Mar 2024 04:30:12 PM UTC. 2024-03-25T18:00:35.673 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-25T18:00:35.673 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T18:00:35.673 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-03-25T18:00:35.674 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T18:00:35.674 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-25T18:00:35.674 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T18:00:35.674 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T18:00:35.674 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-25T18:00:35.674 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-25T18:00:35.674 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-03-25T18:00:35.674 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T18:00:35.674 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 1.3 M 2024-03-25T18:00:35.674 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 5.5 M 2024-03-25T18:00:35.674 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-25T18:00:41.869 INFO:teuthology.orchestra.run.smithi155.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 207 kB/s | 1.3 MB 00:06 2024-03-25T18:00:41.869 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:00:41.870 INFO:teuthology.orchestra.run.smithi155.stdout:Total 207 kB/s | 1.3 MB 00:06 2024-03-25T18:00:41.870 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-25T18:00:41.874 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-25T18:00:41.874 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-25T18:00:41.919 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-25T18:00:41.919 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-25T18:00:42.183 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-25T18:00:42.292 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:00:42.790 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:00:43.063 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:00:43.063 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T18:00:43.063 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-25T18:00:43.063 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:00:43.063 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T18:00:43.063 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-25T18:00:43.256 DEBUG:teuthology.parallel:result is None 2024-03-25T18:04:22.809 INFO:teuthology.orchestra.run.smithi167.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 304 kB/s | 99 MB 05:31 2024-03-25T18:04:22.809 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:22.809 INFO:teuthology.orchestra.run.smithi167.stdout:Total 304 kB/s | 99 MB 05:31 2024-03-25T18:04:22.809 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-03-25T18:04:22.812 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-03-25T18:04:22.812 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-03-25T18:04:23.424 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-03-25T18:04:23.425 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-03-25T18:04:33.737 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-03-25T18:04:33.847 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:04:36.823 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:04:37.074 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:04:37.075 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:04:37.075 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-03-25T18:04:37.075 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:37.075 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:04:37.075 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-03-25T18:04:37.269 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T18:04:37.838 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 1:34:02 ago on Mon 25 Mar 2024 04:30:35 PM UTC. 2024-03-25T18:04:37.989 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-03-25T18:04:37.990 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T18:04:37.990 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-03-25T18:04:37.990 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T18:04:37.990 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-03-25T18:04:37.990 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T18:04:37.990 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:04:37.991 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-03-25T18:04:37.991 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T18:04:37.991 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-03-25T18:04:37.991 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:04:37.991 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 287 M 2024-03-25T18:04:37.991 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 1.2 G 2024-03-25T18:04:37.991 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-03-25T18:17:08.143 INFO:teuthology.orchestra.run.smithi167.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 391 kB/s | 287 MB 12:30 2024-03-25T18:17:08.144 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:17:08.144 INFO:teuthology.orchestra.run.smithi167.stdout:Total 391 kB/s | 287 MB 12:30 2024-03-25T18:17:08.144 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-03-25T18:17:08.146 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-03-25T18:17:08.147 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-03-25T18:17:09.859 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-03-25T18:17:09.859 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-03-25T18:17:39.270 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-03-25T18:17:39.363 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:17:47.046 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:17:47.326 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:17:47.326 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:17:47.326 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-03-25T18:17:47.326 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:17:47.326 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:17:47.326 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-03-25T18:17:47.537 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T18:17:48.105 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 1:47:13 ago on Mon 25 Mar 2024 04:30:35 PM UTC. 2024-03-25T18:17:48.254 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-03-25T18:17:48.255 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T18:17:48.255 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-03-25T18:17:48.255 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T18:17:48.255 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-03-25T18:17:48.256 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T18:17:48.256 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:17:48.256 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-03-25T18:17:48.256 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T18:17:48.256 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-03-25T18:17:48.256 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:17:48.256 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 5.9 M 2024-03-25T18:17:48.256 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 28 M 2024-03-25T18:17:48.256 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-03-25T18:18:03.582 INFO:teuthology.orchestra.run.smithi167.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 397 kB/s | 5.9 MB 00:15 2024-03-25T18:18:03.583 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:18:03.583 INFO:teuthology.orchestra.run.smithi167.stdout:Total 397 kB/s | 5.9 MB 00:15 2024-03-25T18:18:03.583 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-03-25T18:18:03.587 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-03-25T18:18:03.587 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-03-25T18:18:03.660 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-03-25T18:18:03.660 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-03-25T18:18:04.396 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-03-25T18:18:04.505 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:18:05.114 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:18:05.377 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:18:05.377 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:18:05.377 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-03-25T18:18:05.377 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:18:05.377 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:18:05.377 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-03-25T18:18:05.562 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install librados-devel-debuginfo 2024-03-25T18:18:06.127 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 1:47:31 ago on Mon 25 Mar 2024 04:30:35 PM UTC. 2024-03-25T18:18:06.278 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-03-25T18:18:06.279 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T18:18:06.279 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-03-25T18:18:06.279 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T18:18:06.279 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-03-25T18:18:06.279 INFO:teuthology.orchestra.run.smithi167.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T18:18:06.279 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:18:06.279 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-03-25T18:18:06.279 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T18:18:06.279 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-03-25T18:18:06.280 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:18:06.280 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 109 k 2024-03-25T18:18:06.280 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 440 k 2024-03-25T18:18:06.280 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-03-25T18:18:06.968 INFO:teuthology.orchestra.run.smithi167.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 158 kB/s | 109 kB 00:00 2024-03-25T18:18:06.968 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:18:06.969 INFO:teuthology.orchestra.run.smithi167.stdout:Total 158 kB/s | 109 kB 00:00 2024-03-25T18:18:06.969 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-03-25T18:18:06.972 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-03-25T18:18:06.972 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-03-25T18:18:07.008 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-03-25T18:18:07.009 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-03-25T18:18:07.130 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-03-25T18:18:07.226 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:18:07.615 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:18:07.885 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:18:07.885 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:18:07.885 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-03-25T18:18:07.885 INFO:teuthology.orchestra.run.smithi167.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:18:07.885 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:18:07.885 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-03-25T18:18:08.068 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T18:18:08.635 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 1:47:33 ago on Mon 25 Mar 2024 04:30:35 PM UTC. 2024-03-25T18:18:08.786 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-03-25T18:18:08.786 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T18:18:08.787 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-03-25T18:18:08.787 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T18:18:08.787 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-03-25T18:18:08.787 INFO:teuthology.orchestra.run.smithi167.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T18:18:08.787 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:18:08.787 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-03-25T18:18:08.787 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T18:18:08.787 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-03-25T18:18:08.787 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:18:08.787 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 5.4 M 2024-03-25T18:18:08.787 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 25 M 2024-03-25T18:18:08.787 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-03-25T18:18:18.695 INFO:teuthology.orchestra.run.smithi167.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 553 kB/s | 5.4 MB 00:09 2024-03-25T18:18:18.696 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:18:18.696 INFO:teuthology.orchestra.run.smithi167.stdout:Total 553 kB/s | 5.4 MB 00:09 2024-03-25T18:18:18.696 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-03-25T18:18:18.700 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-03-25T18:18:18.701 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-03-25T18:18:18.770 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-03-25T18:18:18.771 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-03-25T18:18:19.440 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-03-25T18:18:19.560 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:18:20.101 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:18:20.375 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:18:20.376 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:18:20.376 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-03-25T18:18:20.376 INFO:teuthology.orchestra.run.smithi167.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:18:20.376 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:18:20.376 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-03-25T18:18:20.603 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install librados2-debuginfo 2024-03-25T18:18:21.171 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 1:47:46 ago on Mon 25 Mar 2024 04:30:35 PM UTC. 2024-03-25T18:18:21.322 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-03-25T18:18:21.322 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T18:18:21.322 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-03-25T18:18:21.323 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T18:18:21.323 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-03-25T18:18:21.323 INFO:teuthology.orchestra.run.smithi167.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T18:18:21.323 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:18:21.323 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-03-25T18:18:21.323 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T18:18:21.323 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-03-25T18:18:21.323 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:18:21.323 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 44 M 2024-03-25T18:18:21.323 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 253 M 2024-03-25T18:18:21.323 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-03-25T18:19:29.757 INFO:teuthology.orchestra.run.smithi167.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 663 kB/s | 44 MB 01:08 2024-03-25T18:19:29.757 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:19:29.757 INFO:teuthology.orchestra.run.smithi167.stdout:Total 663 kB/s | 44 MB 01:08 2024-03-25T18:19:29.758 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-03-25T18:19:29.761 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-03-25T18:19:29.761 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-03-25T18:19:30.062 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-03-25T18:19:30.062 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-03-25T18:19:35.125 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-03-25T18:19:35.213 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:19:37.041 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:19:37.306 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:19:37.306 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:19:37.306 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-03-25T18:19:37.306 INFO:teuthology.orchestra.run.smithi167.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:19:37.307 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:19:37.307 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-03-25T18:19:37.493 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install librbd1-debuginfo 2024-03-25T18:19:38.064 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 1:49:03 ago on Mon 25 Mar 2024 04:30:35 PM UTC. 2024-03-25T18:19:38.216 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-03-25T18:19:38.216 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T18:19:38.217 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-03-25T18:19:38.217 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T18:19:38.217 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-03-25T18:19:38.217 INFO:teuthology.orchestra.run.smithi167.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:19:38.217 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:19:38.217 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-03-25T18:19:38.217 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T18:19:38.217 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-03-25T18:19:38.217 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:19:38.218 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 52 M 2024-03-25T18:19:38.218 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 308 M 2024-03-25T18:19:38.218 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-03-25T18:21:01.334 INFO:teuthology.orchestra.run.smithi167.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 639 kB/s | 52 MB 01:23 2024-03-25T18:21:01.335 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:21:01.335 INFO:teuthology.orchestra.run.smithi167.stdout:Total 639 kB/s | 52 MB 01:23 2024-03-25T18:21:01.335 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-03-25T18:21:01.338 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-03-25T18:21:01.338 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-03-25T18:21:01.685 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-03-25T18:21:01.685 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-03-25T18:21:07.686 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-03-25T18:21:07.915 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:21:10.186 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:21:10.440 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:21:10.441 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:21:10.441 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-03-25T18:21:10.441 INFO:teuthology.orchestra.run.smithi167.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:21:10.441 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:21:10.441 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-03-25T18:21:10.624 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T18:21:11.183 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 1:50:36 ago on Mon 25 Mar 2024 04:30:35 PM UTC. 2024-03-25T18:21:11.333 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-03-25T18:21:11.333 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T18:21:11.333 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-03-25T18:21:11.333 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T18:21:11.333 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-03-25T18:21:11.334 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T18:21:11.334 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:21:11.334 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-03-25T18:21:11.334 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T18:21:11.334 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-03-25T18:21:11.334 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:21:11.335 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 615 k 2024-03-25T18:21:11.335 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 2.1 M 2024-03-25T18:21:11.335 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-03-25T18:21:13.394 INFO:teuthology.orchestra.run.smithi167.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 299 kB/s | 615 kB 00:02 2024-03-25T18:21:13.394 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:21:13.395 INFO:teuthology.orchestra.run.smithi167.stdout:Total 298 kB/s | 615 kB 00:02 2024-03-25T18:21:13.395 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-03-25T18:21:13.399 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-03-25T18:21:13.399 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-03-25T18:21:13.441 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-03-25T18:21:13.441 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-03-25T18:21:13.610 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-03-25T18:21:13.702 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:21:14.116 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:21:14.382 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:21:14.382 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:21:14.382 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-03-25T18:21:14.382 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:21:14.382 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:21:14.382 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-03-25T18:21:14.570 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install python3-rados-debuginfo 2024-03-25T18:21:15.137 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 1:50:40 ago on Mon 25 Mar 2024 04:30:35 PM UTC. 2024-03-25T18:21:15.288 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-03-25T18:21:15.288 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T18:21:15.288 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-03-25T18:21:15.288 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T18:21:15.288 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-03-25T18:21:15.288 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T18:21:15.288 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:21:15.288 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-03-25T18:21:15.288 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T18:21:15.289 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-03-25T18:21:15.289 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:21:15.289 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 771 k 2024-03-25T18:21:15.289 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 2.7 M 2024-03-25T18:21:15.289 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-03-25T18:21:18.039 INFO:teuthology.orchestra.run.smithi167.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 280 kB/s | 771 kB 00:02 2024-03-25T18:21:18.039 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:21:18.039 INFO:teuthology.orchestra.run.smithi167.stdout:Total 280 kB/s | 771 kB 00:02 2024-03-25T18:21:18.040 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-03-25T18:21:18.044 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-03-25T18:21:18.044 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-03-25T18:21:18.085 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-03-25T18:21:18.085 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-03-25T18:21:18.271 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-03-25T18:21:18.380 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:21:18.722 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:21:18.982 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:21:18.982 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:21:18.982 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-03-25T18:21:18.982 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:21:18.982 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:21:18.983 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-03-25T18:21:19.180 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T18:21:19.742 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 1:50:44 ago on Mon 25 Mar 2024 04:30:35 PM UTC. 2024-03-25T18:21:19.892 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-03-25T18:21:19.893 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T18:21:19.893 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-03-25T18:21:19.894 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T18:21:19.894 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-03-25T18:21:19.894 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T18:21:19.894 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:21:19.894 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-03-25T18:21:19.894 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T18:21:19.894 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-03-25T18:21:19.894 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:21:19.894 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 1.1 M 2024-03-25T18:21:19.894 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 4.3 M 2024-03-25T18:21:19.894 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-03-25T18:21:21.942 INFO:teuthology.orchestra.run.smithi167.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 562 kB/s | 1.1 MB 00:02 2024-03-25T18:21:21.942 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:21:21.943 INFO:teuthology.orchestra.run.smithi167.stdout:Total 562 kB/s | 1.1 MB 00:02 2024-03-25T18:21:21.943 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-03-25T18:21:21.947 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-03-25T18:21:21.947 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-03-25T18:21:21.990 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-03-25T18:21:21.990 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-03-25T18:21:22.213 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-03-25T18:21:22.307 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:21:22.696 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:21:22.948 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:21:22.948 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:21:22.948 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-03-25T18:21:22.948 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:21:22.948 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:21:22.948 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-03-25T18:21:23.159 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T18:21:23.720 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 1:50:48 ago on Mon 25 Mar 2024 04:30:35 PM UTC. 2024-03-25T18:21:23.871 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-03-25T18:21:23.872 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T18:21:23.872 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-03-25T18:21:23.872 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T18:21:23.872 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-03-25T18:21:23.872 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T18:21:23.872 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:21:23.872 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-03-25T18:21:23.872 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T18:21:23.872 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-03-25T18:21:23.872 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:21:23.873 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 337 k 2024-03-25T18:21:23.873 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 1.0 M 2024-03-25T18:21:23.873 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-03-25T18:21:25.228 INFO:teuthology.orchestra.run.smithi167.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 249 kB/s | 337 kB 00:01 2024-03-25T18:21:25.228 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:21:25.229 INFO:teuthology.orchestra.run.smithi167.stdout:Total 249 kB/s | 337 kB 00:01 2024-03-25T18:21:25.229 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-03-25T18:21:25.233 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-03-25T18:21:25.233 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-03-25T18:21:25.273 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-03-25T18:21:25.273 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-03-25T18:21:25.416 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-03-25T18:21:25.535 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:21:25.911 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:21:26.171 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:21:26.171 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:21:26.171 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-03-25T18:21:26.171 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:21:26.171 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:21:26.172 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-03-25T18:21:26.374 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T18:21:26.940 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 1:50:51 ago on Mon 25 Mar 2024 04:30:35 PM UTC. 2024-03-25T18:21:27.091 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-03-25T18:21:27.092 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T18:21:27.093 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-03-25T18:21:27.093 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T18:21:27.093 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-03-25T18:21:27.093 INFO:teuthology.orchestra.run.smithi167.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T18:21:27.093 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:21:27.093 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-03-25T18:21:27.093 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T18:21:27.093 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-03-25T18:21:27.093 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:21:27.093 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 509 k 2024-03-25T18:21:27.094 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 2.5 M 2024-03-25T18:21:27.094 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-03-25T18:21:29.098 INFO:teuthology.orchestra.run.smithi167.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 254 kB/s | 509 kB 00:02 2024-03-25T18:21:29.099 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:21:29.099 INFO:teuthology.orchestra.run.smithi167.stdout:Total 254 kB/s | 509 kB 00:02 2024-03-25T18:21:29.100 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-03-25T18:21:29.104 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-03-25T18:21:29.104 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-03-25T18:21:29.145 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-03-25T18:21:29.145 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-03-25T18:21:29.305 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-03-25T18:21:29.406 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:21:29.798 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:21:30.089 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:21:30.089 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:21:30.089 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-03-25T18:21:30.089 INFO:teuthology.orchestra.run.smithi167.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:21:30.089 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:21:30.089 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-03-25T18:21:30.290 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T18:21:30.850 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 1:50:55 ago on Mon 25 Mar 2024 04:30:35 PM UTC. 2024-03-25T18:21:30.998 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-03-25T18:21:30.999 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T18:21:30.999 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-03-25T18:21:30.999 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T18:21:30.999 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-03-25T18:21:30.999 INFO:teuthology.orchestra.run.smithi167.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:21:30.999 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:21:30.999 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-03-25T18:21:30.999 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T18:21:30.999 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-03-25T18:21:30.999 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:21:31.000 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 52 M 2024-03-25T18:21:31.000 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 298 M 2024-03-25T18:21:31.000 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-03-25T18:23:36.146 INFO:teuthology.orchestra.run.smithi167.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 422 kB/s | 52 MB 02:05 2024-03-25T18:23:36.146 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:23:36.146 INFO:teuthology.orchestra.run.smithi167.stdout:Total 422 kB/s | 52 MB 02:05 2024-03-25T18:23:36.147 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-03-25T18:23:36.149 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-03-25T18:23:36.150 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-03-25T18:23:36.484 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-03-25T18:23:36.485 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-03-25T18:23:42.388 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-03-25T18:23:42.484 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:23:44.594 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:23:44.877 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:23:44.877 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:23:44.877 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-03-25T18:23:44.877 INFO:teuthology.orchestra.run.smithi167.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:23:44.877 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:23:44.878 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-03-25T18:23:45.074 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T18:23:45.643 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 1:53:10 ago on Mon 25 Mar 2024 04:30:35 PM UTC. 2024-03-25T18:23:45.792 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-03-25T18:23:45.792 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T18:23:45.792 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-03-25T18:23:45.792 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T18:23:45.793 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-03-25T18:23:45.793 INFO:teuthology.orchestra.run.smithi167.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T18:23:45.793 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:23:45.793 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-03-25T18:23:45.793 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-03-25T18:23:45.793 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-03-25T18:23:45.793 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:23:45.793 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 1.3 M 2024-03-25T18:23:45.794 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 5.5 M 2024-03-25T18:23:45.794 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-03-25T18:23:48.955 INFO:teuthology.orchestra.run.smithi167.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 406 kB/s | 1.3 MB 00:03 2024-03-25T18:23:48.955 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:23:48.956 INFO:teuthology.orchestra.run.smithi167.stdout:Total 405 kB/s | 1.3 MB 00:03 2024-03-25T18:23:48.956 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-03-25T18:23:48.960 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-03-25T18:23:48.960 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-03-25T18:23:49.004 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-03-25T18:23:49.004 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-03-25T18:23:49.241 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-03-25T18:23:49.346 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:23:49.758 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:23:50.027 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:23:50.027 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:23:50.027 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-03-25T18:23:50.027 INFO:teuthology.orchestra.run.smithi167.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:23:50.027 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:23:50.027 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-03-25T18:23:50.249 DEBUG:teuthology.parallel:result is None 2024-03-25T18:23:50.250 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:23:50.393 DEBUG:teuthology.orchestra.run.smithi155:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T18:23:50.418 INFO:teuthology.orchestra.run.smithi155.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T18:23:50.419 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T18:23:50.419 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T18:23:50.421 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:23:50.635 DEBUG:teuthology.orchestra.run.smithi167:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T18:23:50.656 INFO:teuthology.orchestra.run.smithi167.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T18:23:50.656 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T18:23:50.656 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T18:23:50.657 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-25T18:23:50.657 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-25T18:23:50.658 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T18:23:50.690 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-03-25T18:23:50.690 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T18:23:50.723 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-25T18:23:50.723 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-25T18:23:50.724 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T18:23:50.762 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T18:23:50.832 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-03-25T18:23:50.832 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T18:23:50.860 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T18:23:50.927 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-25T18:23:50.927 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-25T18:23:50.927 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T18:23:50.957 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T18:23:51.027 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-03-25T18:23:51.027 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T18:23:51.054 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T18:23:51.121 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-25T18:23:51.121 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-25T18:23:51.122 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T18:23:51.151 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T18:23:51.221 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-03-25T18:23:51.221 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T18:23:51.248 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T18:23:51.316 INFO:teuthology.run_tasks:Running task ceph... 2024-03-25T18:23:51.406 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-25T18:23:51.407 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 777 /var/log/ceph 2024-03-25T18:23:51.409 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod 777 /var/log/ceph 2024-03-25T18:23:51.437 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-25T18:23:51.437 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T18:23:51.480 DEBUG:teuthology.orchestra.run.smithi167:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T18:23:51.507 INFO:tasks.ceph:Creating extra log directories... 2024-03-25T18:23:51.507 DEBUG:teuthology.orchestra.run.smithi155:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T18:23:51.550 DEBUG:teuthology.orchestra.run.smithi167:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T18:23:51.603 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-25T18:23:51.604 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-25T18:23:51.604 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621273', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7621273', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 6785, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi155.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG9GnDhi3qlVC9dImWU7K2yGAWDRaZmta/anqc1QhHI2yQmtpqA6J5ARO+7/Ii0+YsV6tYMdaqBJUklrT1zU160=', 'smithi167.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGkR3F73DIi0cpF43Ju4TRu8aT0j6LuPuo7GMORA5vpzJkhzAQzfFd85n98iFpyhxZ9s35yGY/5pPQTjWg7QfRc='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-25_13:28:33', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-25T18:23:51.604 DEBUG:teuthology.orchestra.run.smithi155:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T18:23:51.619 DEBUG:teuthology.orchestra.run.smithi167:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T18:23:51.634 DEBUG:teuthology.orchestra.run.smithi155:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T18:23:51.675 DEBUG:teuthology.orchestra.run.smithi167:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T18:23:51.701 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-25T18:23:51.701 DEBUG:teuthology.orchestra.run.smithi155:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T18:23:51.757 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T18:23:51.757 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_1 2024-03-25T18:23:51.812 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T18:23:51.813 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:23:51.813 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-25T18:23:51.813 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:23:51.813 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:23:51.813 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2024-03-25 16:36:43.004980267 +0000 2024-03-25T18:23:51.813 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2024-03-25 16:29:25.217420684 +0000 2024-03-25T18:23:51.813 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2024-03-25 16:29:25.217420684 +0000 2024-03-25T18:23:51.813 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: 2024-03-25 16:29:25.217420684 +0000 2024-03-25T18:23:51.813 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T18:23:51.879 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2024-03-25T18:23:51.879 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2024-03-25T18:23:51.879 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000469393 s, 1.1 MB/s 2024-03-25T18:23:51.881 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T18:23:51.936 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_2 2024-03-25T18:23:51.991 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T18:23:51.991 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:23:51.991 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-25T18:23:51.991 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:23:51.991 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:23:51.992 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2024-03-25 16:36:43.004980267 +0000 2024-03-25T18:23:51.992 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2024-03-25 16:29:25.555413702 +0000 2024-03-25T18:23:51.992 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2024-03-25 16:29:25.555413702 +0000 2024-03-25T18:23:51.992 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: 2024-03-25 16:29:25.555413702 +0000 2024-03-25T18:23:51.992 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T18:23:52.057 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2024-03-25T18:23:52.057 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2024-03-25T18:23:52.057 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000176233 s, 2.9 MB/s 2024-03-25T18:23:52.058 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T18:23:52.114 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_3 2024-03-25T18:23:52.168 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T18:23:52.168 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:23:52.168 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-25T18:23:52.168 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:23:52.168 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:23:52.168 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2024-03-25 16:36:43.004980267 +0000 2024-03-25T18:23:52.168 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2024-03-25 16:29:25.901406555 +0000 2024-03-25T18:23:52.168 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2024-03-25 16:29:25.901406555 +0000 2024-03-25T18:23:52.169 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: 2024-03-25 16:29:25.901406555 +0000 2024-03-25T18:23:52.169 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T18:23:52.233 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2024-03-25T18:23:52.233 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2024-03-25T18:23:52.234 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000198224 s, 2.6 MB/s 2024-03-25T18:23:52.235 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T18:23:52.291 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_4 2024-03-25T18:23:52.345 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T18:23:52.345 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:23:52.345 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-25T18:23:52.345 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:23:52.346 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:23:52.346 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2024-03-25 16:36:43.005980246 +0000 2024-03-25T18:23:52.346 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2024-03-25 16:29:26.232399717 +0000 2024-03-25T18:23:52.346 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2024-03-25 16:29:26.232399717 +0000 2024-03-25T18:23:52.346 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: 2024-03-25 16:29:26.232399717 +0000 2024-03-25T18:23:52.346 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T18:23:52.411 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2024-03-25T18:23:52.411 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2024-03-25T18:23:52.411 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000218684 s, 2.3 MB/s 2024-03-25T18:23:52.412 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T18:23:52.468 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:23:52.468 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-03-25T18:23:52.468 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T18:23:52.485 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T18:23:52.485 DEBUG:teuthology.orchestra.run.smithi167:> stat /dev/vg_nvme/lv_1 2024-03-25T18:23:52.541 INFO:teuthology.orchestra.run.smithi167.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T18:23:52.541 INFO:teuthology.orchestra.run.smithi167.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:23:52.541 INFO:teuthology.orchestra.run.smithi167.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-25T18:23:52.541 INFO:teuthology.orchestra.run.smithi167.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:23:52.541 INFO:teuthology.orchestra.run.smithi167.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:23:52.541 INFO:teuthology.orchestra.run.smithi167.stdout:Access: 2024-03-25 16:37:50.029938204 +0000 2024-03-25T18:23:52.541 INFO:teuthology.orchestra.run.smithi167.stdout:Modify: 2024-03-25 16:29:22.822375768 +0000 2024-03-25T18:23:52.541 INFO:teuthology.orchestra.run.smithi167.stdout:Change: 2024-03-25 16:29:22.822375768 +0000 2024-03-25T18:23:52.541 INFO:teuthology.orchestra.run.smithi167.stdout: Birth: 2024-03-25 16:29:22.822375768 +0000 2024-03-25T18:23:52.541 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T18:23:52.608 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records in 2024-03-25T18:23:52.608 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records out 2024-03-25T18:23:52.608 INFO:teuthology.orchestra.run.smithi167.stderr:512 bytes copied, 0.000217091 s, 2.4 MB/s 2024-03-25T18:23:52.609 DEBUG:teuthology.orchestra.run.smithi167:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T18:23:52.666 DEBUG:teuthology.orchestra.run.smithi167:> stat /dev/vg_nvme/lv_2 2024-03-25T18:23:52.722 INFO:teuthology.orchestra.run.smithi167.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T18:23:52.722 INFO:teuthology.orchestra.run.smithi167.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:23:52.722 INFO:teuthology.orchestra.run.smithi167.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-25T18:23:52.722 INFO:teuthology.orchestra.run.smithi167.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:23:52.722 INFO:teuthology.orchestra.run.smithi167.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:23:52.722 INFO:teuthology.orchestra.run.smithi167.stdout:Access: 2024-03-25 16:37:50.030938186 +0000 2024-03-25T18:23:52.722 INFO:teuthology.orchestra.run.smithi167.stdout:Modify: 2024-03-25 16:29:23.165369516 +0000 2024-03-25T18:23:52.722 INFO:teuthology.orchestra.run.smithi167.stdout:Change: 2024-03-25 16:29:23.165369516 +0000 2024-03-25T18:23:52.722 INFO:teuthology.orchestra.run.smithi167.stdout: Birth: 2024-03-25 16:29:23.165369516 +0000 2024-03-25T18:23:52.723 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T18:23:52.788 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records in 2024-03-25T18:23:52.788 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records out 2024-03-25T18:23:52.788 INFO:teuthology.orchestra.run.smithi167.stderr:512 bytes copied, 0.000208248 s, 2.5 MB/s 2024-03-25T18:23:52.789 DEBUG:teuthology.orchestra.run.smithi167:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T18:23:52.845 DEBUG:teuthology.orchestra.run.smithi167:> stat /dev/vg_nvme/lv_3 2024-03-25T18:23:52.901 INFO:teuthology.orchestra.run.smithi167.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T18:23:52.901 INFO:teuthology.orchestra.run.smithi167.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:23:52.901 INFO:teuthology.orchestra.run.smithi167.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-25T18:23:52.901 INFO:teuthology.orchestra.run.smithi167.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:23:52.901 INFO:teuthology.orchestra.run.smithi167.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:23:52.901 INFO:teuthology.orchestra.run.smithi167.stdout:Access: 2024-03-25 16:37:50.030938186 +0000 2024-03-25T18:23:52.901 INFO:teuthology.orchestra.run.smithi167.stdout:Modify: 2024-03-25 16:29:23.499363428 +0000 2024-03-25T18:23:52.901 INFO:teuthology.orchestra.run.smithi167.stdout:Change: 2024-03-25 16:29:23.499363428 +0000 2024-03-25T18:23:52.901 INFO:teuthology.orchestra.run.smithi167.stdout: Birth: 2024-03-25 16:29:23.499363428 +0000 2024-03-25T18:23:52.902 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T18:23:52.966 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records in 2024-03-25T18:23:52.967 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records out 2024-03-25T18:23:52.967 INFO:teuthology.orchestra.run.smithi167.stderr:512 bytes copied, 0.000217022 s, 2.4 MB/s 2024-03-25T18:23:52.968 DEBUG:teuthology.orchestra.run.smithi167:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T18:23:53.023 DEBUG:teuthology.orchestra.run.smithi167:> stat /dev/vg_nvme/lv_4 2024-03-25T18:23:53.077 INFO:teuthology.orchestra.run.smithi167.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T18:23:53.077 INFO:teuthology.orchestra.run.smithi167.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:23:53.077 INFO:teuthology.orchestra.run.smithi167.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-25T18:23:53.077 INFO:teuthology.orchestra.run.smithi167.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:23:53.077 INFO:teuthology.orchestra.run.smithi167.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:23:53.077 INFO:teuthology.orchestra.run.smithi167.stdout:Access: 2024-03-25 16:37:50.031938169 +0000 2024-03-25T18:23:53.077 INFO:teuthology.orchestra.run.smithi167.stdout:Modify: 2024-03-25 16:29:23.844357140 +0000 2024-03-25T18:23:53.077 INFO:teuthology.orchestra.run.smithi167.stdout:Change: 2024-03-25 16:29:23.844357140 +0000 2024-03-25T18:23:53.078 INFO:teuthology.orchestra.run.smithi167.stdout: Birth: 2024-03-25 16:29:23.844357140 +0000 2024-03-25T18:23:53.078 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T18:23:53.142 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records in 2024-03-25T18:23:53.142 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records out 2024-03-25T18:23:53.143 INFO:teuthology.orchestra.run.smithi167.stderr:512 bytes copied, 0.00019985 s, 2.6 MB/s 2024-03-25T18:23:53.144 DEBUG:teuthology.orchestra.run.smithi167:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T18:23:53.199 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:23:53.199 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi155.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi167.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T18:23:53.199 INFO:tasks.ceph:Generating config... 2024-03-25T18:23:53.200 INFO:tasks.ceph:[client] debug monc = 20 2024-03-25T18:23:53.200 INFO:tasks.ceph:[client] debug ms = 1 2024-03-25T18:23:53.200 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-25T18:23:53.200 INFO:tasks.ceph:[client] debug rados = 20 2024-03-25T18:23:53.200 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-25T18:23:53.200 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-03-25T18:23:53.200 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-25T18:23:53.201 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-25T18:23:53.201 INFO:tasks.ceph:[global] ms type = async 2024-03-25T18:23:53.201 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-03-25T18:23:53.201 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-25T18:23:53.201 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-25T18:23:53.201 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-25T18:23:53.201 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-25T18:23:53.201 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-25T18:23:53.201 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-25T18:23:53.201 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-03-25T18:23:53.201 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-25T18:23:53.201 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-25T18:23:53.201 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-25T18:23:53.201 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-25T18:23:53.201 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-25T18:23:53.202 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-25T18:23:53.202 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-25T18:23:53.202 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-25T18:23:53.202 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-03-25T18:23:53.202 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-25T18:23:53.202 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-25T18:23:53.202 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-25T18:23:53.202 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-03-25T18:23:53.202 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-03-25T18:23:53.202 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-25T18:23:53.202 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-25T18:23:53.202 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-25T18:23:53.202 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-25T18:23:53.202 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-25T18:23:53.202 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-25T18:23:53.203 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-25T18:23:53.203 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-03-25T18:23:53.203 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-25T18:23:53.203 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-25T18:23:53.203 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-25T18:23:53.203 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-25T18:23:53.203 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-25T18:23:53.203 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-25T18:23:53.203 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-03-25T18:23:53.203 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-03-25T18:23:53.203 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-25T18:23:53.203 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-25T18:23:53.203 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-25T18:23:53.203 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-25T18:23:53.203 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-25T18:23:53.204 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-25T18:23:53.204 INFO:tasks.ceph:Setting up mon.a... 2024-03-25T18:23:53.204 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-25T18:23:53.266 INFO:teuthology.orchestra.run.smithi155.stdout:creating /etc/ceph/ceph.keyring 2024-03-25T18:23:53.268 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-25T18:23:53.313 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:23:53.340 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.155:6789]'), ('mon.c', '[v1:172.21.15.155:6790]'), ('mon.b', '[v1:172.21.15.167:6789]')] 2024-03-25T18:23:53.340 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.155:6789],[v1:172.21.15.155:6790],[v1:172.21.15.167:6789]', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-25T18:23:53.341 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-25T18:23:53.341 DEBUG:teuthology.orchestra.run.smithi155:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T18:23:53.395 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.155:6789]' --addv c '[v1:172.21.15.155:6790]' --addv b '[v1:172.21.15.167:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:23:53.480 INFO:teuthology.orchestra.run.smithi155.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:23:53.480 INFO:teuthology.orchestra.run.smithi155.stdout:monmaptool: generated fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:23:53.480 INFO:teuthology.orchestra.run.smithi155.stdout:setting min_mon_release = quincy 2024-03-25T18:23:53.480 INFO:teuthology.orchestra.run.smithi155.stdout:epoch 0 2024-03-25T18:23:53.480 INFO:teuthology.orchestra.run.smithi155.stdout:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:23:53.480 INFO:teuthology.orchestra.run.smithi155.stdout:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:23:53.480 INFO:teuthology.orchestra.run.smithi155.stdout:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:23:53.480 INFO:teuthology.orchestra.run.smithi155.stdout:min_mon_release 17 (quincy) 2024-03-25T18:23:53.480 INFO:teuthology.orchestra.run.smithi155.stdout:election_strategy: 3 2024-03-25T18:23:53.480 INFO:teuthology.orchestra.run.smithi155.stdout:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:23:53.480 INFO:teuthology.orchestra.run.smithi155.stdout:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:23:53.481 INFO:teuthology.orchestra.run.smithi155.stdout:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:23:53.481 INFO:teuthology.orchestra.run.smithi155.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-25T18:23:53.482 DEBUG:teuthology.orchestra.run.smithi155:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T18:23:53.496 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b... 2024-03-25T18:23:53.497 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T18:23:53.539 DEBUG:teuthology.orchestra.run.smithi167:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T18:23:53.587 INFO:teuthology.orchestra.run.smithi155.stdout:[global] 2024-03-25T18:23:53.587 INFO:teuthology.orchestra.run.smithi155.stdout: chdir = "" 2024-03-25T18:23:53.587 INFO:teuthology.orchestra.run.smithi155.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T18:23:53.587 INFO:teuthology.orchestra.run.smithi155.stdout: auth supported = cephx 2024-03-25T18:23:53.587 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T18:23:53.587 INFO:teuthology.orchestra.run.smithi155.stdout: filestore xattr use omap = true 2024-03-25T18:23:53.587 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T18:23:53.588 INFO:teuthology.orchestra.run.smithi155.stdout: mon clock drift allowed = 1.000 2024-03-25T18:23:53.588 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T18:23:53.588 INFO:teuthology.orchestra.run.smithi155.stdout: osd crush chooseleaf type = 0 2024-03-25T18:23:53.588 INFO:teuthology.orchestra.run.smithi155.stdout: auth debug = true 2024-03-25T18:23:53.588 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T18:23:53.588 INFO:teuthology.orchestra.run.smithi155.stdout: ms die on old message = true 2024-03-25T18:23:53.588 INFO:teuthology.orchestra.run.smithi155.stdout: ms die on bug = true 2024-03-25T18:23:53.588 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T18:23:53.588 INFO:teuthology.orchestra.run.smithi155.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T18:23:53.588 INFO:teuthology.orchestra.run.smithi155.stdout: mon pg warn max object skew = 0 2024-03-25T18:23:53.588 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T18:23:53.588 INFO:teuthology.orchestra.run.smithi155.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T18:23:53.588 INFO:teuthology.orchestra.run.smithi155.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T18:23:53.588 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T18:23:53.588 INFO:teuthology.orchestra.run.smithi155.stdout: osd pool default size = 2 2024-03-25T18:23:53.589 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T18:23:53.589 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd allow primary affinity = true 2024-03-25T18:23:53.589 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd allow pg remap = true 2024-03-25T18:23:53.589 INFO:teuthology.orchestra.run.smithi155.stdout: mon warn on legacy crush tunables = false 2024-03-25T18:23:53.589 INFO:teuthology.orchestra.run.smithi155.stdout: mon warn on crush straw calc version zero = false 2024-03-25T18:23:53.589 INFO:teuthology.orchestra.run.smithi155.stdout: mon warn on no sortbitwise = false 2024-03-25T18:23:53.589 INFO:teuthology.orchestra.run.smithi155.stdout: mon warn on osd down out interval zero = false 2024-03-25T18:23:53.589 INFO:teuthology.orchestra.run.smithi155.stdout: mon warn on too few osds = false 2024-03-25T18:23:53.589 INFO:teuthology.orchestra.run.smithi155.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T18:23:53.589 INFO:teuthology.orchestra.run.smithi155.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T18:23:53.589 INFO:teuthology.orchestra.run.smithi155.stdout: mon_allow_pool_size_one = true 2024-03-25T18:23:53.589 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T18:23:53.589 INFO:teuthology.orchestra.run.smithi155.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T18:23:53.589 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T18:23:53.590 INFO:teuthology.orchestra.run.smithi155.stdout: osd default data pool replay window = 5 2024-03-25T18:23:53.590 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T18:23:53.590 INFO:teuthology.orchestra.run.smithi155.stdout: mon allow pool delete = true 2024-03-25T18:23:53.590 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T18:23:53.590 INFO:teuthology.orchestra.run.smithi155.stdout: mon cluster log file level = debug 2024-03-25T18:23:53.590 INFO:teuthology.orchestra.run.smithi155.stdout: debug asserts on shutdown = true 2024-03-25T18:23:53.590 INFO:teuthology.orchestra.run.smithi155.stdout: mon health detail to clog = false 2024-03-25T18:23:53.590 INFO:teuthology.orchestra.run.smithi155.stdout: mon host = "[v1:172.21.15.155:6789],[v1:172.21.15.155:6790],[v1:172.21.15.167:6789]" 2024-03-25T18:23:53.590 INFO:teuthology.orchestra.run.smithi155.stdout: mon client directed command retry = 5 2024-03-25T18:23:53.590 INFO:teuthology.orchestra.run.smithi155.stdout: mon election default strategy = 3 2024-03-25T18:23:53.590 INFO:teuthology.orchestra.run.smithi155.stdout: ms bind msgr2 = False 2024-03-25T18:23:53.590 INFO:teuthology.orchestra.run.smithi155.stdout: ms inject socket failures = 5000 2024-03-25T18:23:53.590 INFO:teuthology.orchestra.run.smithi155.stdout: ms type = async 2024-03-25T18:23:53.590 INFO:teuthology.orchestra.run.smithi155.stdout: osd heartbeat grace = 80 2024-03-25T18:23:53.591 INFO:teuthology.orchestra.run.smithi155.stdout: fsid = 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:23:53.591 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T18:23:53.591 INFO:teuthology.orchestra.run.smithi155.stdout:[osd] 2024-03-25T18:23:53.591 INFO:teuthology.orchestra.run.smithi155.stdout: osd journal size = 100 2024-03-25T18:23:53.591 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T18:23:53.591 INFO:teuthology.orchestra.run.smithi155.stdout: osd scrub load threshold = 5.0 2024-03-25T18:23:53.591 INFO:teuthology.orchestra.run.smithi155.stdout: osd scrub max interval = 600 2024-03-25T18:23:53.591 INFO:teuthology.orchestra.run.smithi155.stdout: osd mclock profile = high_recovery_ops 2024-03-25T18:23:53.591 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T18:23:53.591 INFO:teuthology.orchestra.run.smithi155.stdout: osd recover clone overlap = true 2024-03-25T18:23:53.591 INFO:teuthology.orchestra.run.smithi155.stdout: osd recovery max chunk = 1048576 2024-03-25T18:23:53.591 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T18:23:53.591 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug shutdown = true 2024-03-25T18:23:53.591 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug op order = true 2024-03-25T18:23:53.591 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug verify stray on activate = true 2024-03-25T18:23:53.592 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug trim objects = true 2024-03-25T18:23:53.592 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T18:23:53.592 INFO:teuthology.orchestra.run.smithi155.stdout: osd open classes on start = true 2024-03-25T18:23:53.592 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug pg log writeout = true 2024-03-25T18:23:53.592 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T18:23:53.592 INFO:teuthology.orchestra.run.smithi155.stdout: osd deep scrub update digest min age = 30 2024-03-25T18:23:53.592 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T18:23:53.592 INFO:teuthology.orchestra.run.smithi155.stdout: osd map max advance = 10 2024-03-25T18:23:53.592 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T18:23:53.592 INFO:teuthology.orchestra.run.smithi155.stdout: journal zero on create = true 2024-03-25T18:23:53.592 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T18:23:53.592 INFO:teuthology.orchestra.run.smithi155.stdout: filestore ondisk finisher threads = 3 2024-03-25T18:23:53.592 INFO:teuthology.orchestra.run.smithi155.stdout: filestore apply finisher threads = 3 2024-03-25T18:23:53.592 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T18:23:53.592 INFO:teuthology.orchestra.run.smithi155.stdout: bdev debug aio = true 2024-03-25T18:23:53.592 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug misdirected ops = true 2024-03-25T18:23:53.593 INFO:teuthology.orchestra.run.smithi155.stdout: bdev async discard = True 2024-03-25T18:23:53.593 INFO:teuthology.orchestra.run.smithi155.stdout: bdev enable discard = True 2024-03-25T18:23:53.593 INFO:teuthology.orchestra.run.smithi155.stdout: bluestore allocator = stupid 2024-03-25T18:23:53.593 INFO:teuthology.orchestra.run.smithi155.stdout: bluestore block size = 96636764160 2024-03-25T18:23:53.593 INFO:teuthology.orchestra.run.smithi155.stdout: bluestore fsck on mount = True 2024-03-25T18:23:53.593 INFO:teuthology.orchestra.run.smithi155.stdout: bluestore zero block detection = True 2024-03-25T18:23:53.593 INFO:teuthology.orchestra.run.smithi155.stdout: debug bluefs = 1 2024-03-25T18:23:53.593 INFO:teuthology.orchestra.run.smithi155.stdout: debug bluestore = 1 2024-03-25T18:23:53.593 INFO:teuthology.orchestra.run.smithi155.stdout: debug monc = 20 2024-03-25T18:23:53.593 INFO:teuthology.orchestra.run.smithi155.stdout: debug ms = 1 2024-03-25T18:23:53.593 INFO:teuthology.orchestra.run.smithi155.stdout: debug osd = 20 2024-03-25T18:23:53.593 INFO:teuthology.orchestra.run.smithi155.stdout: debug rocksdb = 10 2024-03-25T18:23:53.593 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T18:23:53.593 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd full ratio = 0.9 2024-03-25T18:23:53.593 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd nearfull ratio = 0.8 2024-03-25T18:23:53.593 INFO:teuthology.orchestra.run.smithi155.stdout: osd blocked scrub grace period = 3600 2024-03-25T18:23:53.594 INFO:teuthology.orchestra.run.smithi155.stdout: osd class default list = * 2024-03-25T18:23:53.594 INFO:teuthology.orchestra.run.smithi155.stdout: osd class load list = * 2024-03-25T18:23:53.594 INFO:teuthology.orchestra.run.smithi155.stdout: osd client watch timeout = 120 2024-03-25T18:23:53.594 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug verify cached snaps = True 2024-03-25T18:23:53.594 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug verify missing on start = True 2024-03-25T18:23:53.594 INFO:teuthology.orchestra.run.smithi155.stdout: osd failsafe full ratio = 0.95 2024-03-25T18:23:53.594 INFO:teuthology.orchestra.run.smithi155.stdout: osd fast shutdown = False 2024-03-25T18:23:53.594 INFO:teuthology.orchestra.run.smithi155.stdout: osd max markdown count = 1000 2024-03-25T18:23:53.594 INFO:teuthology.orchestra.run.smithi155.stdout: osd mclock override recovery settings = True 2024-03-25T18:23:53.594 INFO:teuthology.orchestra.run.smithi155.stdout: osd objectstore = bluestore 2024-03-25T18:23:53.594 INFO:teuthology.orchestra.run.smithi155.stdout: osd op queue = debug_random 2024-03-25T18:23:53.594 INFO:teuthology.orchestra.run.smithi155.stdout: osd op queue cut off = debug_random 2024-03-25T18:23:53.594 INFO:teuthology.orchestra.run.smithi155.stdout: osd shutdown pgref assert = True 2024-03-25T18:23:53.594 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T18:23:53.594 INFO:teuthology.orchestra.run.smithi155.stdout:[mgr] 2024-03-25T18:23:53.595 INFO:teuthology.orchestra.run.smithi155.stdout: debug ms = 1 2024-03-25T18:23:53.595 INFO:teuthology.orchestra.run.smithi155.stdout: debug mgr = 20 2024-03-25T18:23:53.595 INFO:teuthology.orchestra.run.smithi155.stdout: debug mon = 20 2024-03-25T18:23:53.595 INFO:teuthology.orchestra.run.smithi155.stdout: debug auth = 20 2024-03-25T18:23:53.595 INFO:teuthology.orchestra.run.smithi155.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:23:53.595 INFO:teuthology.orchestra.run.smithi155.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:23:53.595 INFO:teuthology.orchestra.run.smithi155.stdout: mgr/telemetry/nag = false 2024-03-25T18:23:53.595 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T18:23:53.595 INFO:teuthology.orchestra.run.smithi155.stdout:[mon] 2024-03-25T18:23:53.595 INFO:teuthology.orchestra.run.smithi155.stdout: debug ms = 1 2024-03-25T18:23:53.595 INFO:teuthology.orchestra.run.smithi155.stdout: debug mon = 20 2024-03-25T18:23:53.595 INFO:teuthology.orchestra.run.smithi155.stdout: debug paxos = 20 2024-03-25T18:23:53.595 INFO:teuthology.orchestra.run.smithi155.stdout: debug auth = 20 2024-03-25T18:23:53.595 INFO:teuthology.orchestra.run.smithi155.stdout: mon data avail warn = 5 2024-03-25T18:23:53.595 INFO:teuthology.orchestra.run.smithi155.stdout: mon mgr mkfs grace = 240 2024-03-25T18:23:53.595 INFO:teuthology.orchestra.run.smithi155.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:23:53.596 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd reporter subtree level = osd 2024-03-25T18:23:53.596 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd prime pg temp = true 2024-03-25T18:23:53.596 INFO:teuthology.orchestra.run.smithi155.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:23:53.596 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T18:23:53.596 INFO:teuthology.orchestra.run.smithi155.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T18:23:53.596 INFO:teuthology.orchestra.run.smithi155.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T18:23:53.596 INFO:teuthology.orchestra.run.smithi155.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T18:23:53.596 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T18:23:53.596 INFO:teuthology.orchestra.run.smithi155.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T18:23:53.596 INFO:teuthology.orchestra.run.smithi155.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T18:23:53.596 INFO:teuthology.orchestra.run.smithi155.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T18:23:53.596 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T18:23:53.596 INFO:teuthology.orchestra.run.smithi155.stdout: # 1m isn't quite enough 2024-03-25T18:23:53.596 INFO:teuthology.orchestra.run.smithi155.stdout: mon_down_mkfs_grace = 2m 2024-03-25T18:23:53.596 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T18:23:53.597 INFO:teuthology.orchestra.run.smithi155.stdout: mon_warn_on_filestore_osds = false 2024-03-25T18:23:53.597 INFO:teuthology.orchestra.run.smithi155.stdout: mon min osdmap epochs = 50 2024-03-25T18:23:53.597 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd crush smoke test = False 2024-03-25T18:23:53.597 INFO:teuthology.orchestra.run.smithi155.stdout: mon osdmap full prune interval = 2 2024-03-25T18:23:53.597 INFO:teuthology.orchestra.run.smithi155.stdout: mon osdmap full prune min = 15 2024-03-25T18:23:53.597 INFO:teuthology.orchestra.run.smithi155.stdout: mon osdmap full prune txsize = 2 2024-03-25T18:23:53.597 INFO:teuthology.orchestra.run.smithi155.stdout: mon scrub interval = 300 2024-03-25T18:23:53.597 INFO:teuthology.orchestra.run.smithi155.stdout: mon warn on pool no app = False 2024-03-25T18:23:53.597 INFO:teuthology.orchestra.run.smithi155.stdout: paxos service trim min = 10 2024-03-25T18:23:53.597 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T18:23:53.597 INFO:teuthology.orchestra.run.smithi155.stdout:[client] 2024-03-25T18:23:53.597 INFO:teuthology.orchestra.run.smithi155.stdout: rgw cache enabled = true 2024-03-25T18:23:53.597 INFO:teuthology.orchestra.run.smithi155.stdout: rgw enable ops log = true 2024-03-25T18:23:53.597 INFO:teuthology.orchestra.run.smithi155.stdout: rgw enable usage log = true 2024-03-25T18:23:53.597 INFO:teuthology.orchestra.run.smithi155.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T18:23:53.597 INFO:teuthology.orchestra.run.smithi155.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T18:23:53.597 INFO:teuthology.orchestra.run.smithi155.stdout: debug monc = 20 2024-03-25T18:23:53.598 INFO:teuthology.orchestra.run.smithi155.stdout: debug ms = 1 2024-03-25T18:23:53.598 INFO:teuthology.orchestra.run.smithi155.stdout: debug objecter = 20 2024-03-25T18:23:53.598 INFO:teuthology.orchestra.run.smithi155.stdout: debug rados = 20 2024-03-25T18:23:53.598 INFO:teuthology.orchestra.run.smithi155.stdout:[mon.a] 2024-03-25T18:23:53.598 INFO:teuthology.orchestra.run.smithi155.stdout:[mon.c] 2024-03-25T18:23:53.598 INFO:teuthology.orchestra.run.smithi155.stdout:[mon.b] 2024-03-25T18:23:53.598 INFO:teuthology.orchestra.run.smithi167.stdout:[global] 2024-03-25T18:23:53.598 INFO:teuthology.orchestra.run.smithi167.stdout: chdir = "" 2024-03-25T18:23:53.598 INFO:teuthology.orchestra.run.smithi167.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T18:23:53.599 INFO:teuthology.orchestra.run.smithi167.stdout: auth supported = cephx 2024-03-25T18:23:53.599 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:23:53.599 INFO:teuthology.orchestra.run.smithi167.stdout: filestore xattr use omap = true 2024-03-25T18:23:53.599 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:23:53.599 INFO:teuthology.orchestra.run.smithi167.stdout: mon clock drift allowed = 1.000 2024-03-25T18:23:53.599 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:23:53.599 INFO:teuthology.orchestra.run.smithi167.stdout: osd crush chooseleaf type = 0 2024-03-25T18:23:53.599 INFO:teuthology.orchestra.run.smithi167.stdout: auth debug = true 2024-03-25T18:23:53.599 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:23:53.599 INFO:teuthology.orchestra.run.smithi167.stdout: ms die on old message = true 2024-03-25T18:23:53.599 INFO:teuthology.orchestra.run.smithi167.stdout: ms die on bug = true 2024-03-25T18:23:53.599 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:23:53.599 INFO:teuthology.orchestra.run.smithi167.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T18:23:53.599 INFO:teuthology.orchestra.run.smithi167.stdout: mon pg warn max object skew = 0 2024-03-25T18:23:53.599 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:23:53.600 INFO:teuthology.orchestra.run.smithi167.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T18:23:53.600 INFO:teuthology.orchestra.run.smithi167.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T18:23:53.600 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:23:53.600 INFO:teuthology.orchestra.run.smithi167.stdout: osd pool default size = 2 2024-03-25T18:23:53.600 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:23:53.600 INFO:teuthology.orchestra.run.smithi167.stdout: mon osd allow primary affinity = true 2024-03-25T18:23:53.600 INFO:teuthology.orchestra.run.smithi167.stdout: mon osd allow pg remap = true 2024-03-25T18:23:53.600 INFO:teuthology.orchestra.run.smithi167.stdout: mon warn on legacy crush tunables = false 2024-03-25T18:23:53.600 INFO:teuthology.orchestra.run.smithi167.stdout: mon warn on crush straw calc version zero = false 2024-03-25T18:23:53.600 INFO:teuthology.orchestra.run.smithi167.stdout: mon warn on no sortbitwise = false 2024-03-25T18:23:53.600 INFO:teuthology.orchestra.run.smithi167.stdout: mon warn on osd down out interval zero = false 2024-03-25T18:23:53.600 INFO:teuthology.orchestra.run.smithi167.stdout: mon warn on too few osds = false 2024-03-25T18:23:53.600 INFO:teuthology.orchestra.run.smithi167.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T18:23:53.600 INFO:teuthology.orchestra.run.smithi167.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T18:23:53.600 INFO:teuthology.orchestra.run.smithi167.stdout: mon_allow_pool_size_one = true 2024-03-25T18:23:53.601 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:23:53.601 INFO:teuthology.orchestra.run.smithi167.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T18:23:53.601 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:23:53.601 INFO:teuthology.orchestra.run.smithi167.stdout: osd default data pool replay window = 5 2024-03-25T18:23:53.601 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:23:53.601 INFO:teuthology.orchestra.run.smithi167.stdout: mon allow pool delete = true 2024-03-25T18:23:53.601 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:23:53.601 INFO:teuthology.orchestra.run.smithi167.stdout: mon cluster log file level = debug 2024-03-25T18:23:53.601 INFO:teuthology.orchestra.run.smithi167.stdout: debug asserts on shutdown = true 2024-03-25T18:23:53.601 INFO:teuthology.orchestra.run.smithi167.stdout: mon health detail to clog = false 2024-03-25T18:23:53.601 INFO:teuthology.orchestra.run.smithi167.stdout: mon host = "[v1:172.21.15.155:6789],[v1:172.21.15.155:6790],[v1:172.21.15.167:6789]" 2024-03-25T18:23:53.601 INFO:teuthology.orchestra.run.smithi167.stdout: mon client directed command retry = 5 2024-03-25T18:23:53.601 INFO:teuthology.orchestra.run.smithi167.stdout: mon election default strategy = 3 2024-03-25T18:23:53.601 INFO:teuthology.orchestra.run.smithi167.stdout: ms bind msgr2 = False 2024-03-25T18:23:53.601 INFO:teuthology.orchestra.run.smithi167.stdout: ms inject socket failures = 5000 2024-03-25T18:23:53.601 INFO:teuthology.orchestra.run.smithi167.stdout: ms type = async 2024-03-25T18:23:53.602 INFO:teuthology.orchestra.run.smithi167.stdout: osd heartbeat grace = 80 2024-03-25T18:23:53.602 INFO:teuthology.orchestra.run.smithi167.stdout: fsid = 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:23:53.602 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:23:53.602 INFO:teuthology.orchestra.run.smithi167.stdout:[osd] 2024-03-25T18:23:53.602 INFO:teuthology.orchestra.run.smithi167.stdout: osd journal size = 100 2024-03-25T18:23:53.602 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:23:53.602 INFO:teuthology.orchestra.run.smithi167.stdout: osd scrub load threshold = 5.0 2024-03-25T18:23:53.602 INFO:teuthology.orchestra.run.smithi167.stdout: osd scrub max interval = 600 2024-03-25T18:23:53.602 INFO:teuthology.orchestra.run.smithi167.stdout: osd mclock profile = high_recovery_ops 2024-03-25T18:23:53.602 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:23:53.602 INFO:teuthology.orchestra.run.smithi167.stdout: osd recover clone overlap = true 2024-03-25T18:23:53.602 INFO:teuthology.orchestra.run.smithi167.stdout: osd recovery max chunk = 1048576 2024-03-25T18:23:53.602 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:23:53.602 INFO:teuthology.orchestra.run.smithi167.stdout: osd debug shutdown = true 2024-03-25T18:23:53.602 INFO:teuthology.orchestra.run.smithi167.stdout: osd debug op order = true 2024-03-25T18:23:53.603 INFO:teuthology.orchestra.run.smithi167.stdout: osd debug verify stray on activate = true 2024-03-25T18:23:53.603 INFO:teuthology.orchestra.run.smithi167.stdout: osd debug trim objects = true 2024-03-25T18:23:53.603 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:23:53.603 INFO:teuthology.orchestra.run.smithi167.stdout: osd open classes on start = true 2024-03-25T18:23:53.603 INFO:teuthology.orchestra.run.smithi167.stdout: osd debug pg log writeout = true 2024-03-25T18:23:53.603 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:23:53.603 INFO:teuthology.orchestra.run.smithi167.stdout: osd deep scrub update digest min age = 30 2024-03-25T18:23:53.603 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:23:53.603 INFO:teuthology.orchestra.run.smithi167.stdout: osd map max advance = 10 2024-03-25T18:23:53.603 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:23:53.603 INFO:teuthology.orchestra.run.smithi167.stdout: journal zero on create = true 2024-03-25T18:23:53.603 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:23:53.603 INFO:teuthology.orchestra.run.smithi167.stdout: filestore ondisk finisher threads = 3 2024-03-25T18:23:53.603 INFO:teuthology.orchestra.run.smithi167.stdout: filestore apply finisher threads = 3 2024-03-25T18:23:53.603 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:23:53.604 INFO:teuthology.orchestra.run.smithi167.stdout: bdev debug aio = true 2024-03-25T18:23:53.604 INFO:teuthology.orchestra.run.smithi167.stdout: osd debug misdirected ops = true 2024-03-25T18:23:53.604 INFO:teuthology.orchestra.run.smithi167.stdout: bdev async discard = True 2024-03-25T18:23:53.604 INFO:teuthology.orchestra.run.smithi167.stdout: bdev enable discard = True 2024-03-25T18:23:53.604 INFO:teuthology.orchestra.run.smithi167.stdout: bluestore allocator = stupid 2024-03-25T18:23:53.604 INFO:teuthology.orchestra.run.smithi167.stdout: bluestore block size = 96636764160 2024-03-25T18:23:53.604 INFO:teuthology.orchestra.run.smithi167.stdout: bluestore fsck on mount = True 2024-03-25T18:23:53.604 INFO:teuthology.orchestra.run.smithi167.stdout: bluestore zero block detection = True 2024-03-25T18:23:53.604 INFO:teuthology.orchestra.run.smithi167.stdout: debug bluefs = 1 2024-03-25T18:23:53.604 INFO:teuthology.orchestra.run.smithi167.stdout: debug bluestore = 1 2024-03-25T18:23:53.604 INFO:teuthology.orchestra.run.smithi167.stdout: debug monc = 20 2024-03-25T18:23:53.604 INFO:teuthology.orchestra.run.smithi167.stdout: debug ms = 1 2024-03-25T18:23:53.604 INFO:teuthology.orchestra.run.smithi167.stdout: debug osd = 20 2024-03-25T18:23:53.604 INFO:teuthology.orchestra.run.smithi167.stdout: debug rocksdb = 10 2024-03-25T18:23:53.604 INFO:teuthology.orchestra.run.smithi167.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T18:23:53.605 INFO:teuthology.orchestra.run.smithi167.stdout: mon osd full ratio = 0.9 2024-03-25T18:23:53.605 INFO:teuthology.orchestra.run.smithi167.stdout: mon osd nearfull ratio = 0.8 2024-03-25T18:23:53.605 INFO:teuthology.orchestra.run.smithi167.stdout: osd blocked scrub grace period = 3600 2024-03-25T18:23:53.605 INFO:teuthology.orchestra.run.smithi167.stdout: osd class default list = * 2024-03-25T18:23:53.605 INFO:teuthology.orchestra.run.smithi167.stdout: osd class load list = * 2024-03-25T18:23:53.605 INFO:teuthology.orchestra.run.smithi167.stdout: osd client watch timeout = 120 2024-03-25T18:23:53.605 INFO:teuthology.orchestra.run.smithi167.stdout: osd debug verify cached snaps = True 2024-03-25T18:23:53.605 INFO:teuthology.orchestra.run.smithi167.stdout: osd debug verify missing on start = True 2024-03-25T18:23:53.605 INFO:teuthology.orchestra.run.smithi167.stdout: osd failsafe full ratio = 0.95 2024-03-25T18:23:53.605 INFO:teuthology.orchestra.run.smithi167.stdout: osd fast shutdown = False 2024-03-25T18:23:53.605 INFO:teuthology.orchestra.run.smithi167.stdout: osd max markdown count = 1000 2024-03-25T18:23:53.605 INFO:teuthology.orchestra.run.smithi167.stdout: osd mclock override recovery settings = True 2024-03-25T18:23:53.605 INFO:teuthology.orchestra.run.smithi167.stdout: osd objectstore = bluestore 2024-03-25T18:23:53.605 INFO:teuthology.orchestra.run.smithi167.stdout: osd op queue = debug_random 2024-03-25T18:23:53.605 INFO:teuthology.orchestra.run.smithi167.stdout: osd op queue cut off = debug_random 2024-03-25T18:23:53.606 INFO:teuthology.orchestra.run.smithi167.stdout: osd shutdown pgref assert = True 2024-03-25T18:23:53.606 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:23:53.606 INFO:teuthology.orchestra.run.smithi167.stdout:[mgr] 2024-03-25T18:23:53.606 INFO:teuthology.orchestra.run.smithi167.stdout: debug ms = 1 2024-03-25T18:23:53.606 INFO:teuthology.orchestra.run.smithi167.stdout: debug mgr = 20 2024-03-25T18:23:53.606 INFO:teuthology.orchestra.run.smithi167.stdout: debug mon = 20 2024-03-25T18:23:53.606 INFO:teuthology.orchestra.run.smithi167.stdout: debug auth = 20 2024-03-25T18:23:53.606 INFO:teuthology.orchestra.run.smithi167.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:23:53.606 INFO:teuthology.orchestra.run.smithi167.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:23:53.606 INFO:teuthology.orchestra.run.smithi167.stdout: mgr/telemetry/nag = false 2024-03-25T18:23:53.606 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:23:53.606 INFO:teuthology.orchestra.run.smithi167.stdout:[mon] 2024-03-25T18:23:53.606 INFO:teuthology.orchestra.run.smithi167.stdout: debug ms = 1 2024-03-25T18:23:53.606 INFO:teuthology.orchestra.run.smithi167.stdout: debug mon = 20 2024-03-25T18:23:53.606 INFO:teuthology.orchestra.run.smithi167.stdout: debug paxos = 20 2024-03-25T18:23:53.607 INFO:teuthology.orchestra.run.smithi167.stdout: debug auth = 20 2024-03-25T18:23:53.607 INFO:teuthology.orchestra.run.smithi167.stdout: mon data avail warn = 5 2024-03-25T18:23:53.607 INFO:teuthology.orchestra.run.smithi167.stdout: mon mgr mkfs grace = 240 2024-03-25T18:23:53.607 INFO:teuthology.orchestra.run.smithi167.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:23:53.607 INFO:teuthology.orchestra.run.smithi167.stdout: mon osd reporter subtree level = osd 2024-03-25T18:23:53.607 INFO:teuthology.orchestra.run.smithi167.stdout: mon osd prime pg temp = true 2024-03-25T18:23:53.607 INFO:teuthology.orchestra.run.smithi167.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:23:53.607 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:23:53.607 INFO:teuthology.orchestra.run.smithi167.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T18:23:53.607 INFO:teuthology.orchestra.run.smithi167.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T18:23:53.607 INFO:teuthology.orchestra.run.smithi167.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T18:23:53.607 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:23:53.607 INFO:teuthology.orchestra.run.smithi167.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T18:23:53.607 INFO:teuthology.orchestra.run.smithi167.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T18:23:53.607 INFO:teuthology.orchestra.run.smithi167.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T18:23:53.607 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:23:53.608 INFO:teuthology.orchestra.run.smithi167.stdout: # 1m isn't quite enough 2024-03-25T18:23:53.608 INFO:teuthology.orchestra.run.smithi167.stdout: mon_down_mkfs_grace = 2m 2024-03-25T18:23:53.608 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:23:53.608 INFO:teuthology.orchestra.run.smithi167.stdout: mon_warn_on_filestore_osds = false 2024-03-25T18:23:53.608 INFO:teuthology.orchestra.run.smithi167.stdout: mon min osdmap epochs = 50 2024-03-25T18:23:53.608 INFO:teuthology.orchestra.run.smithi167.stdout: mon osd crush smoke test = False 2024-03-25T18:23:53.608 INFO:teuthology.orchestra.run.smithi167.stdout: mon osdmap full prune interval = 2 2024-03-25T18:23:53.608 INFO:teuthology.orchestra.run.smithi167.stdout: mon osdmap full prune min = 15 2024-03-25T18:23:53.608 INFO:teuthology.orchestra.run.smithi167.stdout: mon osdmap full prune txsize = 2 2024-03-25T18:23:53.608 INFO:teuthology.orchestra.run.smithi167.stdout: mon scrub interval = 300 2024-03-25T18:23:53.608 INFO:teuthology.orchestra.run.smithi167.stdout: mon warn on pool no app = False 2024-03-25T18:23:53.608 INFO:teuthology.orchestra.run.smithi167.stdout: paxos service trim min = 10 2024-03-25T18:23:53.608 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-03-25T18:23:53.608 INFO:teuthology.orchestra.run.smithi167.stdout:[client] 2024-03-25T18:23:53.609 INFO:teuthology.orchestra.run.smithi167.stdout: rgw cache enabled = true 2024-03-25T18:23:53.609 INFO:teuthology.orchestra.run.smithi167.stdout: rgw enable ops log = true 2024-03-25T18:23:53.609 INFO:teuthology.orchestra.run.smithi167.stdout: rgw enable usage log = true 2024-03-25T18:23:53.609 INFO:teuthology.orchestra.run.smithi167.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T18:23:53.609 INFO:teuthology.orchestra.run.smithi167.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T18:23:53.609 INFO:teuthology.orchestra.run.smithi167.stdout: debug monc = 20 2024-03-25T18:23:53.609 INFO:teuthology.orchestra.run.smithi167.stdout: debug ms = 1 2024-03-25T18:23:53.609 INFO:teuthology.orchestra.run.smithi167.stdout: debug objecter = 20 2024-03-25T18:23:53.609 INFO:teuthology.orchestra.run.smithi167.stdout: debug rados = 20 2024-03-25T18:23:53.609 INFO:teuthology.orchestra.run.smithi167.stdout:[mon.a] 2024-03-25T18:23:53.609 INFO:teuthology.orchestra.run.smithi167.stdout:[mon.c] 2024-03-25T18:23:53.609 INFO:teuthology.orchestra.run.smithi167.stdout:[mon.b] 2024-03-25T18:23:53.610 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-25T18:23:53.611 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-25T18:23:53.665 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-25T18:23:53.666 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-25T18:23:53.666 DEBUG:teuthology.orchestra.run.smithi155:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-25T18:23:53.680 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-25T18:23:53.680 DEBUG:teuthology.orchestra.run.smithi155:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-25T18:23:53.735 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi155.front.sepia.ceph.com 2024-03-25T18:23:53.735 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-25T18:23:53.735 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T18:23:53.735 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:23:53.814 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-25T18:23:53.814 DEBUG:teuthology.orchestra.run.smithi155:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:23:53.868 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi167.front.sepia.ceph.com 2024-03-25T18:23:53.868 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-03-25T18:23:53.868 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T18:23:53.868 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:23:53.908 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-03-25T18:23:53.908 DEBUG:teuthology.orchestra.run.smithi167:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:23:53.962 INFO:tasks.ceph:Setting up mon nodes... 2024-03-25T18:23:53.962 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-25T18:23:53.963 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T18:23:54.017 INFO:teuthology.orchestra.run.smithi155.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T18:23:54.020 DEBUG:teuthology.orchestra.run.smithi167:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T18:23:54.092 INFO:teuthology.orchestra.run.smithi167.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T18:23:54.094 INFO:tasks.ceph:Setting up mds nodes... 2024-03-25T18:23:54.094 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-25T18:23:54.095 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-25T18:23:54.137 INFO:teuthology.orchestra.run.smithi155.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-25T18:23:54.152 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-25T18:23:54.194 INFO:teuthology.orchestra.run.smithi167.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-25T18:23:54.209 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-25T18:23:54.209 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi155.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi167.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T18:23:54.209 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-25T18:23:54.236 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:23:54.236 INFO:tasks.ceph:role: osd.0 2024-03-25T18:23:54.236 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi155.front.sepia.ceph.com 2024-03-25T18:23:54.236 DEBUG:teuthology.orchestra.run.smithi155:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T18:23:54.310 INFO:teuthology.orchestra.run.smithi155.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:23:54.311 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:23:54.311 INFO:teuthology.orchestra.run.smithi155.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:23:54.311 INFO:teuthology.orchestra.run.smithi155.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:23:54.311 INFO:teuthology.orchestra.run.smithi155.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:23:54.311 INFO:teuthology.orchestra.run.smithi155.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:23:54.311 INFO:teuthology.orchestra.run.smithi155.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:23:54.311 INFO:teuthology.orchestra.run.smithi155.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:23:54.311 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:23:54.311 INFO:teuthology.orchestra.run.smithi155.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:23:54.816 INFO:teuthology.orchestra.run.smithi155.stdout:Discarding blocks...Done. 2024-03-25T18:23:54.818 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi155.front.sepia.ceph.com -o noatime 2024-03-25T18:23:54.818 DEBUG:teuthology.orchestra.run.smithi155:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-25T18:23:54.951 DEBUG:teuthology.orchestra.run.smithi155:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-25T18:23:54.983 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-25T18:23:55.050 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:23:55.050 INFO:tasks.ceph:role: osd.1 2024-03-25T18:23:55.050 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi155.front.sepia.ceph.com 2024-03-25T18:23:55.050 DEBUG:teuthology.orchestra.run.smithi155:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T18:23:55.124 INFO:teuthology.orchestra.run.smithi155.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:23:55.124 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:23:55.124 INFO:teuthology.orchestra.run.smithi155.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:23:55.124 INFO:teuthology.orchestra.run.smithi155.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:23:55.124 INFO:teuthology.orchestra.run.smithi155.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:23:55.125 INFO:teuthology.orchestra.run.smithi155.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:23:55.125 INFO:teuthology.orchestra.run.smithi155.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:23:55.125 INFO:teuthology.orchestra.run.smithi155.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:23:55.125 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:23:55.125 INFO:teuthology.orchestra.run.smithi155.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:23:55.571 INFO:teuthology.orchestra.run.smithi155.stdout:Discarding blocks...Done. 2024-03-25T18:23:55.573 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi155.front.sepia.ceph.com -o noatime 2024-03-25T18:23:55.573 DEBUG:teuthology.orchestra.run.smithi155:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-25T18:23:55.656 DEBUG:teuthology.orchestra.run.smithi155:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-25T18:23:55.728 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-25T18:23:55.796 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:23:55.796 INFO:tasks.ceph:role: osd.2 2024-03-25T18:23:55.796 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi155.front.sepia.ceph.com 2024-03-25T18:23:55.796 DEBUG:teuthology.orchestra.run.smithi155:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T18:23:55.871 INFO:teuthology.orchestra.run.smithi155.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:23:55.871 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:23:55.871 INFO:teuthology.orchestra.run.smithi155.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:23:55.871 INFO:teuthology.orchestra.run.smithi155.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:23:55.871 INFO:teuthology.orchestra.run.smithi155.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:23:55.871 INFO:teuthology.orchestra.run.smithi155.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:23:55.871 INFO:teuthology.orchestra.run.smithi155.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:23:55.871 INFO:teuthology.orchestra.run.smithi155.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:23:55.871 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:23:55.872 INFO:teuthology.orchestra.run.smithi155.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:23:56.379 INFO:teuthology.orchestra.run.smithi155.stdout:Discarding blocks...Done. 2024-03-25T18:23:56.382 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi155.front.sepia.ceph.com -o noatime 2024-03-25T18:23:56.382 DEBUG:teuthology.orchestra.run.smithi155:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-25T18:23:56.500 DEBUG:teuthology.orchestra.run.smithi155:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-25T18:23:56.531 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-25T18:23:56.598 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:23:56.598 INFO:tasks.ceph:role: osd.3 2024-03-25T18:23:56.598 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi155.front.sepia.ceph.com 2024-03-25T18:23:56.599 DEBUG:teuthology.orchestra.run.smithi155:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T18:23:56.673 INFO:teuthology.orchestra.run.smithi155.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:23:56.673 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:23:56.673 INFO:teuthology.orchestra.run.smithi155.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:23:56.673 INFO:teuthology.orchestra.run.smithi155.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:23:56.674 INFO:teuthology.orchestra.run.smithi155.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:23:56.674 INFO:teuthology.orchestra.run.smithi155.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:23:56.674 INFO:teuthology.orchestra.run.smithi155.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:23:56.674 INFO:teuthology.orchestra.run.smithi155.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:23:56.674 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:23:56.674 INFO:teuthology.orchestra.run.smithi155.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:23:57.187 INFO:teuthology.orchestra.run.smithi155.stdout:Discarding blocks...Done. 2024-03-25T18:23:57.189 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi155.front.sepia.ceph.com -o noatime 2024-03-25T18:23:57.189 DEBUG:teuthology.orchestra.run.smithi155:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-25T18:23:57.328 DEBUG:teuthology.orchestra.run.smithi155:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-25T18:23:57.362 DEBUG:teuthology.orchestra.run.smithi155:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:23:57.467 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:23:57.466+0000 7fbafcf9a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-25T18:23:57.467 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:23:57.466+0000 7fbafcf9a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-25T18:23:57.467 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:23:57.466+0000 7fbafcf9a640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:23:57.467 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:23:57.466+0000 7fbafcf9a640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:23:57.468 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:23:57.466+0000 7fbafcf9a640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:23:57.468 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:23:57.467+0000 7fbafcf9a640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-25T18:24:00.314 DEBUG:teuthology.orchestra.run.smithi155:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-25T18:24:00.346 DEBUG:teuthology.orchestra.run.smithi155:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:24:00.439 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:00.437+0000 7fc46238c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-25T18:24:00.439 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:00.437+0000 7fc46238c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-25T18:24:00.439 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:00.437+0000 7fc46238c640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:24:00.439 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:00.437+0000 7fc46238c640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:24:00.439 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:00.437+0000 7fc46238c640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:24:00.439 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:00.437+0000 7fc46238c640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-25T18:24:03.282 DEBUG:teuthology.orchestra.run.smithi155:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-25T18:24:03.313 DEBUG:teuthology.orchestra.run.smithi155:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:24:03.405 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:03.404+0000 7f5a90106640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-25T18:24:03.405 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:03.404+0000 7f5a90106640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-25T18:24:03.406 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:03.404+0000 7f5a90106640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:24:03.406 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:03.404+0000 7f5a90106640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:24:03.406 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:03.404+0000 7f5a90106640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:24:03.406 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:03.405+0000 7f5a90106640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-25T18:24:06.248 DEBUG:teuthology.orchestra.run.smithi155:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-25T18:24:06.279 DEBUG:teuthology.orchestra.run.smithi155:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:24:06.372 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:06.371+0000 7fe006401640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-25T18:24:06.372 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:06.371+0000 7fe006401640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-25T18:24:06.372 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:06.371+0000 7fe006401640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:24:06.373 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:06.371+0000 7fe006401640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:24:06.373 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:06.371+0000 7fe006401640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:24:06.373 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:06.372+0000 7fe006401640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-25T18:24:09.213 DEBUG:teuthology.orchestra.run.smithi155:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-25T18:24:09.245 DEBUG:teuthology.orchestra.run.smithi167:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-25T18:24:09.277 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:24:09.277 INFO:tasks.ceph:role: osd.4 2024-03-25T18:24:09.277 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi167.front.sepia.ceph.com 2024-03-25T18:24:09.277 DEBUG:teuthology.orchestra.run.smithi167:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T18:24:09.356 INFO:teuthology.orchestra.run.smithi167.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:24:09.356 INFO:teuthology.orchestra.run.smithi167.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:24:09.356 INFO:teuthology.orchestra.run.smithi167.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:24:09.356 INFO:teuthology.orchestra.run.smithi167.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:24:09.356 INFO:teuthology.orchestra.run.smithi167.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:24:09.356 INFO:teuthology.orchestra.run.smithi167.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:24:09.356 INFO:teuthology.orchestra.run.smithi167.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:24:09.356 INFO:teuthology.orchestra.run.smithi167.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:24:09.356 INFO:teuthology.orchestra.run.smithi167.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:24:09.356 INFO:teuthology.orchestra.run.smithi167.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:24:09.870 INFO:teuthology.orchestra.run.smithi167.stdout:Discarding blocks...Done. 2024-03-25T18:24:09.872 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi167.front.sepia.ceph.com -o noatime 2024-03-25T18:24:09.872 DEBUG:teuthology.orchestra.run.smithi167:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-25T18:24:10.032 DEBUG:teuthology.orchestra.run.smithi167:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-25T18:24:10.071 DEBUG:teuthology.orchestra.run.smithi167:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-25T18:24:10.142 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:24:10.142 INFO:tasks.ceph:role: osd.5 2024-03-25T18:24:10.142 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi167.front.sepia.ceph.com 2024-03-25T18:24:10.142 DEBUG:teuthology.orchestra.run.smithi167:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T18:24:10.218 INFO:teuthology.orchestra.run.smithi167.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:24:10.218 INFO:teuthology.orchestra.run.smithi167.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:24:10.218 INFO:teuthology.orchestra.run.smithi167.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:24:10.218 INFO:teuthology.orchestra.run.smithi167.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:24:10.219 INFO:teuthology.orchestra.run.smithi167.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:24:10.219 INFO:teuthology.orchestra.run.smithi167.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:24:10.219 INFO:teuthology.orchestra.run.smithi167.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:24:10.219 INFO:teuthology.orchestra.run.smithi167.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:24:10.219 INFO:teuthology.orchestra.run.smithi167.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:24:10.219 INFO:teuthology.orchestra.run.smithi167.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:24:10.666 INFO:teuthology.orchestra.run.smithi167.stdout:Discarding blocks...Done. 2024-03-25T18:24:10.668 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi167.front.sepia.ceph.com -o noatime 2024-03-25T18:24:10.668 DEBUG:teuthology.orchestra.run.smithi167:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-25T18:24:10.765 DEBUG:teuthology.orchestra.run.smithi167:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-25T18:24:10.797 DEBUG:teuthology.orchestra.run.smithi167:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-25T18:24:10.864 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:24:10.864 INFO:tasks.ceph:role: osd.6 2024-03-25T18:24:10.864 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi167.front.sepia.ceph.com 2024-03-25T18:24:10.864 DEBUG:teuthology.orchestra.run.smithi167:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T18:24:10.940 INFO:teuthology.orchestra.run.smithi167.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:24:10.940 INFO:teuthology.orchestra.run.smithi167.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:24:10.940 INFO:teuthology.orchestra.run.smithi167.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:24:10.940 INFO:teuthology.orchestra.run.smithi167.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:24:10.940 INFO:teuthology.orchestra.run.smithi167.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:24:10.940 INFO:teuthology.orchestra.run.smithi167.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:24:10.940 INFO:teuthology.orchestra.run.smithi167.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:24:10.940 INFO:teuthology.orchestra.run.smithi167.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:24:10.940 INFO:teuthology.orchestra.run.smithi167.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:24:10.941 INFO:teuthology.orchestra.run.smithi167.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:24:11.428 INFO:teuthology.orchestra.run.smithi167.stdout:Discarding blocks...Done. 2024-03-25T18:24:11.430 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi167.front.sepia.ceph.com -o noatime 2024-03-25T18:24:11.430 DEBUG:teuthology.orchestra.run.smithi167:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-25T18:24:11.571 DEBUG:teuthology.orchestra.run.smithi167:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-25T18:24:11.604 DEBUG:teuthology.orchestra.run.smithi167:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-25T18:24:11.671 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:24:11.671 INFO:tasks.ceph:role: osd.7 2024-03-25T18:24:11.671 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi167.front.sepia.ceph.com 2024-03-25T18:24:11.671 DEBUG:teuthology.orchestra.run.smithi167:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T18:24:11.744 INFO:teuthology.orchestra.run.smithi167.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:24:11.744 INFO:teuthology.orchestra.run.smithi167.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:24:11.744 INFO:teuthology.orchestra.run.smithi167.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:24:11.745 INFO:teuthology.orchestra.run.smithi167.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:24:11.745 INFO:teuthology.orchestra.run.smithi167.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:24:11.745 INFO:teuthology.orchestra.run.smithi167.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:24:11.745 INFO:teuthology.orchestra.run.smithi167.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:24:11.745 INFO:teuthology.orchestra.run.smithi167.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:24:11.745 INFO:teuthology.orchestra.run.smithi167.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:24:11.745 INFO:teuthology.orchestra.run.smithi167.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:24:12.256 INFO:teuthology.orchestra.run.smithi167.stdout:Discarding blocks...Done. 2024-03-25T18:24:12.258 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi167.front.sepia.ceph.com -o noatime 2024-03-25T18:24:12.258 DEBUG:teuthology.orchestra.run.smithi167:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-25T18:24:12.390 DEBUG:teuthology.orchestra.run.smithi167:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-25T18:24:12.423 DEBUG:teuthology.orchestra.run.smithi167:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:24:12.530 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:12.529+0000 7fbe5e00c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-25T18:24:12.531 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:12.530+0000 7fbe5e00c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-25T18:24:12.531 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:12.530+0000 7fbe5e00c640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:24:12.531 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:12.530+0000 7fbe5e00c640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:24:12.531 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:12.530+0000 7fbe5e00c640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:24:12.531 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:12.530+0000 7fbe5e00c640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-25T18:24:15.378 DEBUG:teuthology.orchestra.run.smithi167:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-25T18:24:15.409 DEBUG:teuthology.orchestra.run.smithi167:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:24:15.500 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:15.499+0000 7f8a55f81640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-25T18:24:15.500 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:15.499+0000 7f8a55f81640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-25T18:24:15.500 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:15.499+0000 7f8a55f81640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:24:15.501 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:15.499+0000 7f8a55f81640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:24:15.501 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:15.499+0000 7f8a55f81640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:24:15.501 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:15.500+0000 7f8a55f81640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-25T18:24:18.335 DEBUG:teuthology.orchestra.run.smithi167:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-25T18:24:18.364 DEBUG:teuthology.orchestra.run.smithi167:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:24:18.452 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:18.451+0000 7fc22480d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-25T18:24:18.452 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:18.451+0000 7fc22480d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-25T18:24:18.452 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:18.451+0000 7fc22480d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:24:18.452 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:18.451+0000 7fc22480d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:24:18.453 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:18.451+0000 7fc22480d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:24:18.453 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:18.452+0000 7fc22480d640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-25T18:24:21.308 DEBUG:teuthology.orchestra.run.smithi167:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-25T18:24:21.338 DEBUG:teuthology.orchestra.run.smithi167:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:24:21.427 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:21.426+0000 7f931778f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-25T18:24:21.428 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:21.426+0000 7f931778f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-25T18:24:21.428 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:21.426+0000 7f931778f640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:24:21.428 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:21.426+0000 7f931778f640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:24:21.428 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:21.426+0000 7f931778f640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:24:21.428 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:21.426+0000 7f931778f640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-25T18:24:24.267 DEBUG:teuthology.orchestra.run.smithi167:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-25T18:24:24.298 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-25T18:24:24.298 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-25T18:24:24.298 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-25T18:24:24.330 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-25T18:24:24.330 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-25T18:24:24.401 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-25T18:24:24.401 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-25T18:24:24.469 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-25T18:24:24.470 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-25T18:24:24.538 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-25T18:24:24.538 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-25T18:24:24.607 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-03-25T18:24:24.607 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-25T18:24:24.639 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-03-25T18:24:24.640 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-25T18:24:24.711 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-03-25T18:24:24.711 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-25T18:24:24.780 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-03-25T18:24:24.780 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-25T18:24:24.850 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-03-25T18:24:24.850 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-25T18:24:24.918 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-25T18:24:24.918 DEBUG:teuthology.orchestra.run.smithi155:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-25T18:24:24.935 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-03-25T18:24:24.935 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-25T18:24:24.974 INFO:tasks.ceph:Adding keys to all mons... 2024-03-25T18:24:24.974 DEBUG:teuthology.orchestra.run.smithi155:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T18:24:24.977 DEBUG:teuthology.orchestra.run.smithi167:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T18:24:25.016 INFO:teuthology.orchestra.run.smithi155.stdout:[mgr.y] 2024-03-25T18:24:25.017 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQA6wQFm9+oMARAAkXzxuvuoycQrS+FDUQvzPQ== 2024-03-25T18:24:25.017 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.0] 2024-03-25T18:24:25.017 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQA9wQFmpXbZGxAA6yXQJTA6iBZXRp/5NDj0Cg== 2024-03-25T18:24:25.017 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.1] 2024-03-25T18:24:25.017 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQBAwQFmgFYqGhAAKXSjmsqDY+nrSyMmqDAngA== 2024-03-25T18:24:25.017 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.2] 2024-03-25T18:24:25.017 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQBDwQFmG5ksGBAAk8yRmwT8F1ACk6q4jw/+kQ== 2024-03-25T18:24:25.017 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.3] 2024-03-25T18:24:25.017 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQBGwQFmPU40FhAArgL84uusr+SSUFEzuWDLQg== 2024-03-25T18:24:25.017 INFO:teuthology.orchestra.run.smithi155.stdout:[mgr.x] 2024-03-25T18:24:25.017 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQA6wQFm2ruBBRAAq5FeiYXfSshB35yJuCTiZQ== 2024-03-25T18:24:25.017 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.4] 2024-03-25T18:24:25.018 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQBMwQFm9FahHxAAJACZLbz+En49A5xmvouXmA== 2024-03-25T18:24:25.018 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.5] 2024-03-25T18:24:25.018 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQBPwQFmbtzRHRAADqp/iiA5TRTtmIzfy9cjFg== 2024-03-25T18:24:25.018 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.6] 2024-03-25T18:24:25.018 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQBSwQFm2uT1GhAANzLyOqblRKPfSmsKRMHd2w== 2024-03-25T18:24:25.018 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.7] 2024-03-25T18:24:25.018 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQBVwQFmWdByGRAAEdz0kfg+31sC6daRs5xttg== 2024-03-25T18:24:25.018 INFO:teuthology.orchestra.run.smithi155.stdout:[client.0] 2024-03-25T18:24:25.018 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQA6wQFmvAsxCBAAdQhhb0s/GSHzni0Q5phRog== 2024-03-25T18:24:25.018 INFO:teuthology.orchestra.run.smithi155.stdout:[client.1] 2024-03-25T18:24:25.018 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQA6wQFmgCabCxAAJLBz2jDD0kv+tQGkuUFK4Q== 2024-03-25T18:24:25.039 INFO:teuthology.orchestra.run.smithi167.stdout:[mgr.y] 2024-03-25T18:24:25.039 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQA6wQFm9+oMARAAkXzxuvuoycQrS+FDUQvzPQ== 2024-03-25T18:24:25.039 INFO:teuthology.orchestra.run.smithi167.stdout:[osd.0] 2024-03-25T18:24:25.039 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQA9wQFmpXbZGxAA6yXQJTA6iBZXRp/5NDj0Cg== 2024-03-25T18:24:25.039 INFO:teuthology.orchestra.run.smithi167.stdout:[osd.1] 2024-03-25T18:24:25.039 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQBAwQFmgFYqGhAAKXSjmsqDY+nrSyMmqDAngA== 2024-03-25T18:24:25.040 INFO:teuthology.orchestra.run.smithi167.stdout:[osd.2] 2024-03-25T18:24:25.040 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQBDwQFmG5ksGBAAk8yRmwT8F1ACk6q4jw/+kQ== 2024-03-25T18:24:25.040 INFO:teuthology.orchestra.run.smithi167.stdout:[osd.3] 2024-03-25T18:24:25.040 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQBGwQFmPU40FhAArgL84uusr+SSUFEzuWDLQg== 2024-03-25T18:24:25.040 INFO:teuthology.orchestra.run.smithi167.stdout:[mgr.x] 2024-03-25T18:24:25.040 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQA6wQFm2ruBBRAAq5FeiYXfSshB35yJuCTiZQ== 2024-03-25T18:24:25.040 INFO:teuthology.orchestra.run.smithi167.stdout:[osd.4] 2024-03-25T18:24:25.040 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQBMwQFm9FahHxAAJACZLbz+En49A5xmvouXmA== 2024-03-25T18:24:25.040 INFO:teuthology.orchestra.run.smithi167.stdout:[osd.5] 2024-03-25T18:24:25.040 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQBPwQFmbtzRHRAADqp/iiA5TRTtmIzfy9cjFg== 2024-03-25T18:24:25.040 INFO:teuthology.orchestra.run.smithi167.stdout:[osd.6] 2024-03-25T18:24:25.040 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQBSwQFm2uT1GhAANzLyOqblRKPfSmsKRMHd2w== 2024-03-25T18:24:25.040 INFO:teuthology.orchestra.run.smithi167.stdout:[osd.7] 2024-03-25T18:24:25.040 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQBVwQFmWdByGRAAEdz0kfg+31sC6daRs5xttg== 2024-03-25T18:24:25.041 INFO:teuthology.orchestra.run.smithi167.stdout:[client.0] 2024-03-25T18:24:25.041 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQA6wQFmvAsxCBAAdQhhb0s/GSHzni0Q5phRog== 2024-03-25T18:24:25.041 INFO:teuthology.orchestra.run.smithi167.stdout:[client.1] 2024-03-25T18:24:25.041 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQA6wQFmgCabCxAAJLBz2jDD0kv+tQGkuUFK4Q== 2024-03-25T18:24:25.041 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:24:25.062 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:24:25.134 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:24:25.136 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:24:25.187 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:24:25.189 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:24:25.240 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:24:25.243 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:24:25.294 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:24:25.296 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:24:25.349 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:24:25.351 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:24:25.404 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:24:25.406 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:24:25.457 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:24:25.459 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:24:25.511 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:24:25.514 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:24:25.566 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:24:25.568 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:24:25.619 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:24:25.621 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:24:25.676 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:24:25.679 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:24:25.734 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-25T18:24:25.734 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-25T18:24:25.760 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:24:25.856 DEBUG:teuthology.orchestra.run.smithi155:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-25T18:24:25.883 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-25T18:24:25.950 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:24:26.047 DEBUG:teuthology.orchestra.run.smithi155:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-25T18:24:26.073 DEBUG:teuthology.orchestra.run.smithi167:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-25T18:24:26.100 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:24:26.197 DEBUG:teuthology.orchestra.run.smithi167:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-25T18:24:26.223 DEBUG:teuthology.orchestra.run.smithi155:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:24:26.225 DEBUG:teuthology.orchestra.run.smithi167:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:24:26.278 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-25T18:24:26.278 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:24:26.278 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-25T18:24:26.279 DEBUG:teuthology.orchestra.run.smithi155:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-03-25T18:24:26.281 INFO:tasks.ceph.mon.a:Started 2024-03-25T18:24:26.281 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:24:26.281 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-25T18:24:26.281 DEBUG:teuthology.orchestra.run.smithi155:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-03-25T18:24:26.283 INFO:tasks.ceph.mon.c:Started 2024-03-25T18:24:26.283 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:24:26.284 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-25T18:24:26.284 DEBUG:teuthology.orchestra.run.smithi167:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-03-25T18:24:26.321 INFO:tasks.ceph.mon.b:Started 2024-03-25T18:24:26.321 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-25T18:24:26.321 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-25T18:24:26.322 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-03-25T18:24:26.324 INFO:tasks.ceph.mgr.y:Started 2024-03-25T18:24:26.324 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-25T18:24:26.324 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-03-25T18:24:26.326 INFO:tasks.ceph.mgr.x:Started 2024-03-25T18:24:26.326 INFO:tasks.ceph:Setting crush tunables to default 2024-03-25T18:24:26.326 DEBUG:teuthology.orchestra.run.smithi155:> sudo ceph --cluster ceph osd crush tunables default 2024-03-25T18:24:26.429 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:26.428+0000 7fc79ab6d640 10 monclient: get_monmap_and_config 2024-03-25T18:24:26.429 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:26.428+0000 7fc79ab6d640 10 monclient: build_initial_monmap 2024-03-25T18:24:26.429 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:26.428+0000 7fc79ab6d640 10 monclient: monmap: 2024-03-25T18:24:26.429 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-03-25T18:24:26.429 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:26.429 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-03-25T18:24:26.429028+0000 2024-03-25T18:24:26.429 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-03-25T18:24:26.429028+0000 2024-03-25T18:24:26.429 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:26.429 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 3 2024-03-25T18:24:26.429 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:24:26.429 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:24:26.430 INFO:teuthology.orchestra.run.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:24:26.430 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-03-25T18:24:26.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:26.428+0000 7fc79ab6d640 1 Processor -- start 2024-03-25T18:24:26.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:26.428+0000 7fc79ab6d640 1 -- start start 2024-03-25T18:24:26.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:26.428+0000 7fc79ab6d640 10 monclient: init 2024-03-25T18:24:26.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:26.429+0000 7fc79ab6d640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:26.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:26.429+0000 7fc79ab6d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:24:26.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:26.429+0000 7fc79ab6d640 10 monclient(hunting): picked mon.noname-c con 0x7fc7940a9920 addr v1:172.21.15.167:6789/0 2024-03-25T18:24:26.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:26.429+0000 7fc79ab6d640 10 monclient(hunting): picked mon.noname-a con 0x7fc7940b45a0 addr v1:172.21.15.155:6789/0 2024-03-25T18:24:26.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:26.429+0000 7fc79ab6d640 10 monclient(hunting): picked mon.noname-b con 0x7fc794146450 addr v1:172.21.15.155:6790/0 2024-03-25T18:24:26.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:26.429+0000 7fc79ab6d640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc79406b100 con 0x7fc7940b45a0 2024-03-25T18:24:26.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:26.429+0000 7fc79ab6d640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7940b2940 con 0x7fc7940a9920 2024-03-25T18:24:26.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:26.429+0000 7fc79ab6d640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc79413fec0 con 0x7fc794146450 2024-03-25T18:24:26.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:26.429+0000 7fc79ab6d640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:26.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:26.429+0000 7fc79ab6d640 10 monclient(hunting): authenticate will time out at 7512.739220s 2024-03-25T18:24:26.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:26.429+0000 7fc79910a640 1 -- >> v1:172.21.15.155:6790/0 conn(0x7fc794146450 legacy=0x7fc79414dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6790/0 2024-03-25T18:24:26.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:26.429+0000 7fc798909640 1 -- >> v1:172.21.15.167:6789/0 conn(0x7fc7940a9920 legacy=0x7fc79413f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.167:6789/0 2024-03-25T18:24:26.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:26.429+0000 7fc793fff640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7fc7940b45a0 legacy=0x7fc794145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-03-25T18:24:26.631 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:26.630+0000 7fc793fff640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7fc7940b45a0 legacy=0x7fc794145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-03-25T18:24:26.631 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:26.630+0000 7fc79910a640 1 -- >> v1:172.21.15.155:6790/0 conn(0x7fc794146450 legacy=0x7fc79414dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6790/0 2024-03-25T18:24:26.631 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:26.630+0000 7fc798909640 1 -- >> v1:172.21.15.167:6789/0 conn(0x7fc7940a9920 legacy=0x7fc79413f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.167:6789/0 2024-03-25T18:24:27.031 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:27.031+0000 7fc793fff640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7fc7940b45a0 legacy=0x7fc794145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-03-25T18:24:27.032 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:27.031+0000 7fc79910a640 1 -- >> v1:172.21.15.155:6790/0 conn(0x7fc794146450 legacy=0x7fc79414dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6790/0 2024-03-25T18:24:27.032 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:27.031+0000 7fc798909640 1 -- >> v1:172.21.15.167:6789/0 conn(0x7fc7940a9920 legacy=0x7fc79413f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.167:6789/0 2024-03-25T18:24:27.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:27.832+0000 7fc793fff640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7fc7940b45a0 legacy=0x7fc794145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-03-25T18:24:27.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:27.832+0000 7fc79910a640 1 -- >> v1:172.21.15.155:6790/0 conn(0x7fc794146450 legacy=0x7fc79414dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6790/0 2024-03-25T18:24:27.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:27.832+0000 7fc798909640 1 -- >> v1:172.21.15.167:6789/0 conn(0x7fc7940a9920 legacy=0x7fc79413f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.167:6789/0 2024-03-25T18:24:29.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:29.429+0000 7fc7927fc640 10 monclient(hunting): tick 2024-03-25T18:24:29.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:29.429+0000 7fc7927fc640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:24:29.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:29.429+0000 7fc7927fc640 1 monclient(hunting): continuing hunt 2024-03-25T18:24:29.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:29.429+0000 7fc7927fc640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:24:29.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:29.429+0000 7fc7927fc640 1 -- >> v1:172.21.15.155:6790/0 conn(0x7fc794146450 legacy=0x7fc79414dc70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:24:29.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:29.429+0000 7fc7927fc640 1 -- >> v1:172.21.15.167:6789/0 conn(0x7fc7940a9920 legacy=0x7fc79413f7b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:24:29.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:29.429+0000 7fc7927fc640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7fc7940b45a0 legacy=0x7fc794145d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:24:29.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:29.429+0000 7fc7927fc640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:24:29.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:29.429+0000 7fc7927fc640 10 monclient(hunting): picked mon.noname-c con 0x7fc7940a9920 addr v1:172.21.15.167:6789/0 2024-03-25T18:24:29.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:29.429+0000 7fc7927fc640 10 monclient(hunting): picked mon.noname-a con 0x7fc7940b45a0 addr v1:172.21.15.155:6789/0 2024-03-25T18:24:29.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:29.429+0000 7fc7927fc640 10 monclient(hunting): picked mon.noname-b con 0x7fc794146450 addr v1:172.21.15.155:6790/0 2024-03-25T18:24:29.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:29.429+0000 7fc7927fc640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc79406b100 con 0x7fc7940b45a0 2024-03-25T18:24:29.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:29.429+0000 7fc7927fc640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7940b2940 con 0x7fc7940a9920 2024-03-25T18:24:29.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:29.429+0000 7fc7927fc640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc79413fec0 con 0x7fc794146450 2024-03-25T18:24:29.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:29.429+0000 7fc7927fc640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:29.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:29.429+0000 7fc793fff640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7fc7940b45a0 legacy=0x7fc778008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-03-25T18:24:29.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:29.429+0000 7fc798909640 1 -- >> v1:172.21.15.167:6789/0 conn(0x7fc7940a9920 legacy=0x7fc7780056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.167:6789/0 2024-03-25T18:24:29.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:29.429+0000 7fc79910a640 1 -- >> v1:172.21.15.155:6790/0 conn(0x7fc794146450 legacy=0x7fc77800c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6790/0 2024-03-25T18:24:29.631 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:29.630+0000 7fc793fff640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7fc7940b45a0 legacy=0x7fc778008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-03-25T18:24:29.631 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:29.630+0000 7fc79910a640 1 -- >> v1:172.21.15.155:6790/0 conn(0x7fc794146450 legacy=0x7fc77800c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6790/0 2024-03-25T18:24:29.631 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:29.630+0000 7fc798909640 1 -- >> v1:172.21.15.167:6789/0 conn(0x7fc7940a9920 legacy=0x7fc7780056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.167:6789/0 2024-03-25T18:24:30.032 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:30.031+0000 7fc793fff640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7fc7940b45a0 legacy=0x7fc778008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-03-25T18:24:30.032 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:30.031+0000 7fc79910a640 1 -- >> v1:172.21.15.155:6790/0 conn(0x7fc794146450 legacy=0x7fc77800c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6790/0 2024-03-25T18:24:30.032 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:30.031+0000 7fc798909640 1 -- >> v1:172.21.15.167:6789/0 conn(0x7fc7940a9920 legacy=0x7fc7780056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.167:6789/0 2024-03-25T18:24:30.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:30.832+0000 7fc793fff640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7fc7940b45a0 legacy=0x7fc778008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-03-25T18:24:30.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:30.832+0000 7fc79910a640 1 -- >> v1:172.21.15.155:6790/0 conn(0x7fc794146450 legacy=0x7fc77800c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6790/0 2024-03-25T18:24:30.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:30.832+0000 7fc798909640 1 -- >> v1:172.21.15.167:6789/0 conn(0x7fc7940a9920 legacy=0x7fc7780056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.167:6789/0 2024-03-25T18:24:32.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:32.429+0000 7fc7927fc640 10 monclient(hunting): tick 2024-03-25T18:24:32.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:32.429+0000 7fc7927fc640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:24:32.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:32.429+0000 7fc7927fc640 1 monclient(hunting): continuing hunt 2024-03-25T18:24:32.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:32.429+0000 7fc7927fc640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:24:32.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:32.429+0000 7fc7927fc640 1 -- >> v1:172.21.15.155:6790/0 conn(0x7fc794146450 legacy=0x7fc77800c670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:24:32.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:32.429+0000 7fc7927fc640 1 -- >> v1:172.21.15.167:6789/0 conn(0x7fc7940a9920 legacy=0x7fc7780056d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:24:32.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:32.429+0000 7fc7927fc640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7fc7940b45a0 legacy=0x7fc778008ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:24:32.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:32.429+0000 7fc7927fc640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:24:32.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:32.429+0000 7fc7927fc640 10 monclient(hunting): picked mon.noname-b con 0x7fc794146450 addr v1:172.21.15.155:6790/0 2024-03-25T18:24:32.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:32.429+0000 7fc7927fc640 10 monclient(hunting): picked mon.noname-a con 0x7fc7940b45a0 addr v1:172.21.15.155:6789/0 2024-03-25T18:24:32.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:32.429+0000 7fc7927fc640 10 monclient(hunting): picked mon.noname-c con 0x7fc7940a9920 addr v1:172.21.15.167:6789/0 2024-03-25T18:24:32.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:32.429+0000 7fc7927fc640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc778010e20 con 0x7fc7940b45a0 2024-03-25T18:24:32.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:32.429+0000 7fc7927fc640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc79406b100 con 0x7fc7940a9920 2024-03-25T18:24:32.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:32.429+0000 7fc7927fc640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7940b2940 con 0x7fc794146450 2024-03-25T18:24:32.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:32.429+0000 7fc7927fc640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:32.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:32.429+0000 7fc798909640 1 -- >> v1:172.21.15.155:6790/0 conn(0x7fc794146450 legacy=0x7fc77800fdf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6790/0 2024-03-25T18:24:32.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:32.429+0000 7fc793fff640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7fc7940b45a0 legacy=0x7fc77800f690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-03-25T18:24:32.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:32.429+0000 7fc79910a640 1 -- >> v1:172.21.15.167:6789/0 conn(0x7fc7940a9920 legacy=0x7fc7780056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.167:6789/0 2024-03-25T18:24:32.631 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:32.630+0000 7fc79910a640 1 -- >> v1:172.21.15.167:6789/0 conn(0x7fc7940a9920 legacy=0x7fc7780056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.167:6789/0 2024-03-25T18:24:32.631 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:32.630+0000 7fc798909640 1 -- >> v1:172.21.15.155:6790/0 conn(0x7fc794146450 legacy=0x7fc77800fdf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6790/0 2024-03-25T18:24:32.631 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:32.630+0000 7fc793fff640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7fc7940b45a0 legacy=0x7fc77800f690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-03-25T18:24:33.032 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:33.031+0000 7fc798909640 1 -- >> v1:172.21.15.155:6790/0 conn(0x7fc794146450 legacy=0x7fc77800fdf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6790/0 2024-03-25T18:24:33.032 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:33.031+0000 7fc793fff640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7fc7940b45a0 legacy=0x7fc77800f690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-03-25T18:24:33.032 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:33.031+0000 7fc79910a640 1 -- >> v1:172.21.15.167:6789/0 conn(0x7fc7940a9920 legacy=0x7fc7780056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.167:6789/0 2024-03-25T18:24:33.276 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:00:06.855 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:33.354 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:00:06.959 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:33.355 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:00:06.960 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:33.583 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:00:07.161 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:33.661 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:00:07.266 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:33.664 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:00:07.269 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:33.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:33.832+0000 7fc798909640 1 -- >> v1:172.21.15.155:6790/0 conn(0x7fc794146450 legacy=0x7fc77800fdf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6790/0 2024-03-25T18:24:33.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:33.832+0000 7fc793fff640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7fc7940b45a0 legacy=0x7fc77800f690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-03-25T18:24:33.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:33.832+0000 7fc79910a640 1 -- >> v1:172.21.15.167:6789/0 conn(0x7fc7940a9920 legacy=0x7fc7780056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.167:6789/0 2024-03-25T18:24:35.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:35.429+0000 7fc7927fc640 10 monclient(hunting): tick 2024-03-25T18:24:35.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:35.429+0000 7fc7927fc640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:24:35.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:35.429+0000 7fc7927fc640 1 monclient(hunting): continuing hunt 2024-03-25T18:24:35.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:35.429+0000 7fc7927fc640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:24:35.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:35.429+0000 7fc7927fc640 1 -- >> v1:172.21.15.155:6790/0 conn(0x7fc794146450 legacy=0x7fc77800fdf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:24:35.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:35.429+0000 7fc7927fc640 1 -- >> v1:172.21.15.167:6789/0 conn(0x7fc7940a9920 legacy=0x7fc7780056d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:24:35.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:35.429+0000 7fc7927fc640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7fc7940b45a0 legacy=0x7fc77800f690 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:24:35.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:35.429+0000 7fc7927fc640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:24:35.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:35.429+0000 7fc7927fc640 10 monclient(hunting): picked mon.noname-b con 0x7fc794146450 addr v1:172.21.15.155:6790/0 2024-03-25T18:24:35.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:35.430+0000 7fc7927fc640 10 monclient(hunting): picked mon.noname-c con 0x7fc7940a9920 addr v1:172.21.15.167:6789/0 2024-03-25T18:24:35.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:35.430+0000 7fc7927fc640 10 monclient(hunting): picked mon.noname-a con 0x7fc7940b45a0 addr v1:172.21.15.155:6789/0 2024-03-25T18:24:35.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:35.430+0000 7fc7927fc640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc79413fec0 con 0x7fc7940b45a0 2024-03-25T18:24:35.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:35.430+0000 7fc7927fc640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc778010e20 con 0x7fc7940a9920 2024-03-25T18:24:35.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:35.430+0000 7fc7927fc640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc79406b100 con 0x7fc794146450 2024-03-25T18:24:35.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:35.430+0000 7fc7927fc640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:35.469 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:35.469+0000 7fc79910a640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fc7940b45a0 0x7fc77800f690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:50900/0 (socket says 172.21.15.155:50900) 2024-03-25T18:24:35.469 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:35.469+0000 7fc79910a640 1 -- 172.21.15.155:0/2013319775 learned_addr learned my addr 172.21.15.155:0/2013319775 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:24:38.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:38.430+0000 7fc7927fc640 10 monclient(hunting): tick 2024-03-25T18:24:38.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:38.430+0000 7fc7927fc640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:24:38.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:38.430+0000 7fc7927fc640 1 monclient(hunting): continuing hunt 2024-03-25T18:24:38.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:38.430+0000 7fc7927fc640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:24:38.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:38.430+0000 7fc7927fc640 1 -- 172.21.15.155:0/2013319775 >> v1:172.21.15.155:6790/0 conn(0x7fc794146450 legacy=0x7fc778008e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:38.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:38.430+0000 7fc7927fc640 1 -- 172.21.15.155:0/2013319775 >> v1:172.21.15.167:6789/0 conn(0x7fc7940a9920 legacy=0x7fc77800fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:38.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:38.430+0000 7fc7927fc640 1 -- 172.21.15.155:0/2013319775 >> v1:172.21.15.155:6789/0 conn(0x7fc7940b45a0 legacy=0x7fc77800f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:38.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:38.430+0000 7fc7927fc640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:24:38.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:38.430+0000 7fc7927fc640 10 monclient(hunting): picked mon.noname-b con 0x7fc794146450 addr v1:172.21.15.155:6790/0 2024-03-25T18:24:38.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:38.430+0000 7fc7927fc640 10 monclient(hunting): picked mon.noname-a con 0x7fc7780057a0 addr v1:172.21.15.155:6789/0 2024-03-25T18:24:38.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:38.430+0000 7fc7927fc640 10 monclient(hunting): picked mon.noname-c con 0x7fc7940a9920 addr v1:172.21.15.167:6789/0 2024-03-25T18:24:38.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:38.430+0000 7fc7927fc640 1 -- 172.21.15.155:0/2013319775 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7940b2940 con 0x7fc7780057a0 2024-03-25T18:24:38.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:38.430+0000 7fc7927fc640 1 -- 172.21.15.155:0/2013319775 --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc77800d680 con 0x7fc7940a9920 2024-03-25T18:24:38.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:38.430+0000 7fc7927fc640 1 -- 172.21.15.155:0/2013319775 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc77800e820 con 0x7fc794146450 2024-03-25T18:24:38.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:38.430+0000 7fc7927fc640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:41.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:41.430+0000 7fc7927fc640 10 monclient(hunting): tick 2024-03-25T18:24:41.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:41.430+0000 7fc7927fc640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:24:41.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:41.430+0000 7fc7927fc640 1 monclient(hunting): continuing hunt 2024-03-25T18:24:41.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:41.430+0000 7fc7927fc640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:24:41.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:41.430+0000 7fc7927fc640 1 -- 172.21.15.155:0/2013319775 >> v1:172.21.15.155:6790/0 conn(0x7fc794146450 legacy=0x7fc778008e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:41.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:41.430+0000 7fc7927fc640 1 -- 172.21.15.155:0/2013319775 >> v1:172.21.15.167:6789/0 conn(0x7fc7940a9920 legacy=0x7fc77800fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:41.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:41.430+0000 7fc7927fc640 1 -- 172.21.15.155:0/2013319775 >> v1:172.21.15.155:6789/0 conn(0x7fc7780057a0 legacy=0x7fc778010600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:41.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:41.430+0000 7fc7927fc640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:24:41.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:41.430+0000 7fc7927fc640 10 monclient(hunting): picked mon.noname-b con 0x7fc77800d810 addr v1:172.21.15.155:6790/0 2024-03-25T18:24:41.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:41.430+0000 7fc7927fc640 10 monclient(hunting): picked mon.noname-c con 0x7fc7940a9920 addr v1:172.21.15.167:6789/0 2024-03-25T18:24:41.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:41.430+0000 7fc7927fc640 10 monclient(hunting): picked mon.noname-a con 0x7fc77800e360 addr v1:172.21.15.155:6789/0 2024-03-25T18:24:41.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:41.430+0000 7fc7927fc640 1 -- 172.21.15.155:0/2013319775 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc77800f9c0 con 0x7fc77800e360 2024-03-25T18:24:41.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:41.430+0000 7fc7927fc640 1 -- 172.21.15.155:0/2013319775 --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc778010650 con 0x7fc7940a9920 2024-03-25T18:24:41.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:41.430+0000 7fc7927fc640 1 -- 172.21.15.155:0/2013319775 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc778010880 con 0x7fc77800d810 2024-03-25T18:24:41.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:41.430+0000 7fc7927fc640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:44.000 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:43.999+0000 7fc7937fe640 1 -- 172.21.15.155:0/2013319775 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1434371125 0 0) 0x7fc77800f9c0 con 0x7fc77800e360 2024-03-25T18:24:44.000 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:43.999+0000 7fc7937fe640 10 monclient(hunting): _negotiate 2024-03-25T18:24:44.000 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:43.999+0000 7fc7937fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:44.000 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:43.999+0000 7fc7937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:44.000 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:43.999+0000 7fc7937fe640 10 monclient(hunting): my global_id is 4101 2024-03-25T18:24:44.000 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:43.999+0000 7fc7937fe640 1 -- 172.21.15.155:0/2013319775 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc784003520 con 0x7fc77800e360 2024-03-25T18:24:44.030 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.029+0000 7fc7937fe640 1 -- 172.21.15.155:0/2013319775 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4229645123 0 0) 0x7fc778010880 con 0x7fc77800d810 2024-03-25T18:24:44.030 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.029+0000 7fc7937fe640 10 monclient(hunting): _negotiate 2024-03-25T18:24:44.030 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.029+0000 7fc7937fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:44.030 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.029+0000 7fc7937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:44.030 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.029+0000 7fc7937fe640 10 monclient(hunting): my global_id is 4106 2024-03-25T18:24:44.030 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.029+0000 7fc7937fe640 1 -- 172.21.15.155:0/2013319775 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc77800f9c0 con 0x7fc77800d810 2024-03-25T18:24:44.120 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.119+0000 7fc7937fe640 1 -- 172.21.15.155:0/2013319775 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 356693606 0 0) 0x7fc784003520 con 0x7fc77800e360 2024-03-25T18:24:44.120 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.119+0000 7fc7937fe640 1 -- 172.21.15.155:0/2013319775 >> v1:172.21.15.155:6790/0 conn(0x7fc77800d810 legacy=0x7fc77800dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:44.120 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.119+0000 7fc7937fe640 1 -- 172.21.15.155:0/2013319775 >> v1:172.21.15.167:6789/0 conn(0x7fc7940a9920 legacy=0x7fc778008e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:44.120 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.119+0000 7fc7937fe640 10 monclient: _finish_hunting 0 2024-03-25T18:24:44.120 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.119+0000 7fc7937fe640 1 monclient: found mon.noname-a 2024-03-25T18:24:44.120 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.119+0000 7fc7937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:44.120 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.119+0000 7fc7937fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-25T18:24:44.120 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.119+0000 7fc7937fe640 1 -- 172.21.15.155:0/2013319775 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc778010ab0 con 0x7fc77800e360 2024-03-25T18:24:44.120 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.119+0000 7fc7937fe640 10 monclient: _finish_auth 0 2024-03-25T18:24:44.120 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.119+0000 7fc7937fe640 10 monclient: _check_auth_tickets 2024-03-25T18:24:44.120 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.119+0000 7fc7937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:44.120 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.119+0000 7fc7937fe640 1 -- 172.21.15.155:0/2013319775 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fc78c0042b0 con 0x7fc77800e360 2024-03-25T18:24:44.121 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.119+0000 7fc7937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:44.121 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.119+0000 7fc7937fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:24:44.121 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.119+0000 7fc7937fe640 10 monclient: dump: 2024-03-25T18:24:44.121 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-03-25T18:24:44.121 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:44.121 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:44.121 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:44.121 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:44.121 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 3 2024-03-25T18:24:44.121 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:44.121 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:44.121 INFO:teuthology.orchestra.run.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:44.121 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-03-25T18:24:44.121 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.119+0000 7fc79ab6d640 5 monclient: authenticate success, global_id 4101 2024-03-25T18:24:44.121 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.119+0000 7fc79ab6d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:44.127 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.126+0000 7fc7937fe640 1 -- 172.21.15.155:0/2013319775 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc78c004410 con 0x7fc77800e360 2024-03-25T18:24:44.127 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.126+0000 7fc7937fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:44.127 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.126+0000 7fc7937fe640 1 -- 172.21.15.155:0/2013319775 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fc78c004860 con 0x7fc77800e360 2024-03-25T18:24:44.127 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.126+0000 7fc7937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:44.127 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.126+0000 7fc7937fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:44.127 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.126+0000 7fc7937fe640 10 monclient: dump: 2024-03-25T18:24:44.127 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-03-25T18:24:44.127 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:44.127 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:44.127 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:44.127 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:44.127 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 3 2024-03-25T18:24:44.127 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:44.128 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:44.128 INFO:teuthology.orchestra.run.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:44.128 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-03-25T18:24:44.128 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.126+0000 7fc79ab6d640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:44.128 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.126+0000 7fc79ab6d640 4 set_mon_vals no callback set 2024-03-25T18:24:44.131 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.127+0000 7fc79ab6d640 10 monclient: shutdown 2024-03-25T18:24:44.131 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.127+0000 7fc79ab6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:44.131 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.127+0000 7fc79ab6d640 1 -- 172.21.15.155:0/2013319775 >> v1:172.21.15.155:6789/0 conn(0x7fc77800e360 legacy=0x7fc77800fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:44.131 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.127+0000 7fc79ab6d640 1 -- 172.21.15.155:0/2013319775 shutdown_connections 2024-03-25T18:24:44.131 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.127+0000 7fc79ab6d640 1 -- 172.21.15.155:0/2013319775 >> 172.21.15.155:0/2013319775 conn(0x7fc7940b7f40 msgr2=0x7fc7940b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:44.131 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.127+0000 7fc79ab6d640 1 -- 172.21.15.155:0/2013319775 shutdown_connections 2024-03-25T18:24:44.131 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.127+0000 7fc79ab6d640 1 -- 172.21.15.155:0/2013319775 wait complete. 2024-03-25T18:24:44.131 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.127+0000 7fc79ab6d640 10 monclient: build_initial_monmap 2024-03-25T18:24:44.131 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.127+0000 7fc79ab6d640 10 monclient: monmap: 2024-03-25T18:24:44.131 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-03-25T18:24:44.131 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:44.131 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 0.000000 2024-03-25T18:24:44.131 INFO:teuthology.orchestra.run.smithi155.stderr:created 0.000000 2024-03-25T18:24:44.131 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:44.132 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-03-25T18:24:44.132 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:24:44.132 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:24:44.132 INFO:teuthology.orchestra.run.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:24:44.132 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-03-25T18:24:44.132 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.128+0000 7fc79ab6d640 1 librados: starting msgr at 2024-03-25T18:24:44.132 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.128+0000 7fc79ab6d640 1 librados: starting objecter 2024-03-25T18:24:44.132 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.128+0000 7fc79ab6d640 1 Processor -- start 2024-03-25T18:24:44.132 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.128+0000 7fc79ab6d640 1 -- start start 2024-03-25T18:24:44.132 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.128+0000 7fc79ab6d640 1 librados: setting wanted keys 2024-03-25T18:24:44.132 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.128+0000 7fc79ab6d640 1 librados: calling monclient init 2024-03-25T18:24:44.132 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.128+0000 7fc79ab6d640 10 monclient: init 2024-03-25T18:24:44.132 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.129+0000 7fc79ab6d640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:44.132 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.129+0000 7fc79ab6d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:24:44.132 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.129+0000 7fc79ab6d640 10 monclient(hunting): picked mon.noname-b con 0x7fc77800d810 addr v1:172.21.15.167:6789/0 2024-03-25T18:24:44.133 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.129+0000 7fc79ab6d640 10 monclient(hunting): picked mon.noname-a con 0x7fc77800e360 addr v1:172.21.15.155:6789/0 2024-03-25T18:24:44.133 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.129+0000 7fc79ab6d640 10 monclient(hunting): picked mon.noname-c con 0x7fc7940a9920 addr v1:172.21.15.155:6790/0 2024-03-25T18:24:44.133 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.129+0000 7fc79ab6d640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7941db9a0 con 0x7fc77800e360 2024-03-25T18:24:44.133 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.129+0000 7fc79ab6d640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7941dbb30 con 0x7fc77800d810 2024-03-25T18:24:44.133 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.129+0000 7fc79ab6d640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7940b1140 con 0x7fc7940a9920 2024-03-25T18:24:44.133 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.129+0000 7fc79ab6d640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:44.133 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.129+0000 7fc79ab6d640 10 monclient(hunting): authenticate will time out at 7530.438860s 2024-03-25T18:24:44.133 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.129+0000 7fc79910a640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fc7940a9920 0x7fc7940ace60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:43610/0 (socket says 172.21.15.155:43610) 2024-03-25T18:24:44.133 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.129+0000 7fc79910a640 1 -- 172.21.15.155:0/2107260321 learned_addr learned my addr 172.21.15.155:0/2107260321 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:24:44.133 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.129+0000 7fc7917fa640 10 client.?.objecter ms_handle_connect 0x7fc7940a9920 2024-03-25T18:24:44.133 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.129+0000 7fc7917fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:44.135 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.135+0000 7fc7917fa640 10 client.?.objecter ms_handle_connect 0x7fc77800e360 2024-03-25T18:24:44.135 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.135+0000 7fc7917fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:44.156 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.155+0000 7fc7917fa640 10 client.?.objecter ms_handle_connect 0x7fc77800d810 2024-03-25T18:24:44.156 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.155+0000 7fc7917fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:44.169 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.168+0000 7fc7917fa640 1 -- 172.21.15.155:0/2107260321 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2865889050 0 0) 0x7fc7941db9a0 con 0x7fc77800e360 2024-03-25T18:24:44.169 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.168+0000 7fc7917fa640 10 monclient(hunting): _negotiate 2024-03-25T18:24:44.169 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.168+0000 7fc7917fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:44.169 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.168+0000 7fc7917fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:44.169 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.168+0000 7fc7917fa640 10 monclient(hunting): my global_id is 4113 2024-03-25T18:24:44.169 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.168+0000 7fc7917fa640 1 -- 172.21.15.155:0/2107260321 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc778010880 con 0x7fc77800e360 2024-03-25T18:24:44.235 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.234+0000 7fc7917fa640 1 -- 172.21.15.155:0/2107260321 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 191160689 0 0) 0x7fc7940b1140 con 0x7fc7940a9920 2024-03-25T18:24:44.236 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.234+0000 7fc7917fa640 10 monclient(hunting): _negotiate 2024-03-25T18:24:44.236 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.234+0000 7fc7917fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:44.236 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.234+0000 7fc7917fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:44.236 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.234+0000 7fc7917fa640 10 monclient(hunting): my global_id is 4112 2024-03-25T18:24:44.236 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.234+0000 7fc7917fa640 1 -- 172.21.15.155:0/2107260321 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7941db9a0 con 0x7fc7940a9920 2024-03-25T18:24:44.266 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.265+0000 7fc7917fa640 1 -- 172.21.15.155:0/2107260321 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3406809399 0 0) 0x7fc778010880 con 0x7fc77800e360 2024-03-25T18:24:44.267 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.265+0000 7fc7917fa640 1 -- 172.21.15.155:0/2107260321 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7940b1140 con 0x7fc77800e360 2024-03-25T18:24:44.267 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.265+0000 7fc7917fa640 1 -- 172.21.15.155:0/2107260321 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fc7800042a0 con 0x7fc77800e360 2024-03-25T18:24:44.267 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.265+0000 7fc7917fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:44.267 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.265+0000 7fc7917fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:24:44.267 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.265+0000 7fc7917fa640 10 monclient(hunting): dump: 2024-03-25T18:24:44.267 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-03-25T18:24:44.267 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:44.267 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:44.267 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:44.267 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:44.267 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 3 2024-03-25T18:24:44.267 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:44.267 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:44.267 INFO:teuthology.orchestra.run.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:44.267 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-03-25T18:24:44.268 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.265+0000 7fc7917fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:44.268 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.267+0000 7fc7917fa640 1 -- 172.21.15.155:0/2107260321 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 830665257 0 0) 0x7fc7941dbb30 con 0x7fc77800d810 2024-03-25T18:24:44.268 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.267+0000 7fc7917fa640 10 monclient(hunting): _negotiate 2024-03-25T18:24:44.268 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.267+0000 7fc7917fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:44.268 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.267+0000 7fc7917fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:44.268 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.267+0000 7fc7917fa640 10 monclient(hunting): my global_id is 4105 2024-03-25T18:24:44.268 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.267+0000 7fc7917fa640 1 -- 172.21.15.155:0/2107260321 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc778010880 con 0x7fc77800d810 2024-03-25T18:24:44.277 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.276+0000 7fc7917fa640 1 -- 172.21.15.155:0/2107260321 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2260185178 0 0) 0x7fc7940b1140 con 0x7fc77800e360 2024-03-25T18:24:44.277 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.276+0000 7fc7917fa640 1 -- 172.21.15.155:0/2107260321 >> v1:172.21.15.155:6790/0 conn(0x7fc7940a9920 legacy=0x7fc7940ace60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:44.277 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.276+0000 7fc7917fa640 1 -- 172.21.15.155:0/2107260321 >> v1:172.21.15.167:6789/0 conn(0x7fc77800d810 legacy=0x7fc7941db290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:44.277 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.276+0000 7fc7917fa640 10 monclient: _finish_hunting 0 2024-03-25T18:24:44.277 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.276+0000 7fc7917fa640 1 monclient: found mon.a 2024-03-25T18:24:44.277 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.276+0000 7fc7917fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:44.277 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.276+0000 7fc7917fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:24:44.278 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.276+0000 7fc7917fa640 1 -- 172.21.15.155:0/2107260321 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7940b12d0 con 0x7fc77800e360 2024-03-25T18:24:44.278 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.276+0000 7fc7917fa640 10 monclient: _finish_auth 0 2024-03-25T18:24:44.278 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.276+0000 7fc7917fa640 10 monclient: _check_auth_tickets 2024-03-25T18:24:44.278 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.276+0000 7fc7917fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:44.278 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.276+0000 7fc79ab6d640 5 monclient: authenticate success, global_id 4113 2024-03-25T18:24:44.278 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.276+0000 7fc79ab6d640 10 monclient: _renew_subs 2024-03-25T18:24:44.278 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.276+0000 7fc79ab6d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:24:44.278 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.276+0000 7fc79ab6d640 1 -- 172.21.15.155:0/2107260321 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7941e3570 con 0x7fc77800e360 2024-03-25T18:24:44.278 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.276+0000 7fc79ab6d640 10 client.4113.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:44.278 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.276+0000 7fc79ab6d640 10 monclient: _renew_subs 2024-03-25T18:24:44.278 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.276+0000 7fc79ab6d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:24:44.278 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.276+0000 7fc79ab6d640 1 -- 172.21.15.155:0/2107260321 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7941e38e0 con 0x7fc77800e360 2024-03-25T18:24:44.278 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.276+0000 7fc79ab6d640 1 librados: init done 2024-03-25T18:24:44.279 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.277+0000 7fc79ab6d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:44.279 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.277+0000 7fc79ab6d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:44.279 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.277+0000 7fc79ab6d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:24:44.279 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.277+0000 7fc79ab6d640 1 -- 172.21.15.155:0/2107260321 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7940ac470 con 0x7fc77800e360 2024-03-25T18:24:44.446 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.445+0000 7fc7917fa640 1 -- 172.21.15.155:0/2107260321 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc7800046b0 con 0x7fc77800e360 2024-03-25T18:24:44.447 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.445+0000 7fc7917fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:44.447 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.445+0000 7fc7917fa640 1 -- 172.21.15.155:0/2107260321 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fc780004920 con 0x7fc77800e360 2024-03-25T18:24:44.447 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.445+0000 7fc7917fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:44.447 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.445+0000 7fc7917fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:44.447 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.445+0000 7fc7917fa640 10 monclient: dump: 2024-03-25T18:24:44.447 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-03-25T18:24:44.447 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:44.447 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:44.447 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:44.447 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:44.447 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 3 2024-03-25T18:24:44.447 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:44.447 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:44.448 INFO:teuthology.orchestra.run.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:44.448 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-03-25T18:24:44.448 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.445+0000 7fc7917fa640 1 -- 172.21.15.155:0/2107260321 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fc780004ed0 con 0x7fc77800e360 2024-03-25T18:24:44.448 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.445+0000 7fc792ffd640 4 set_mon_vals no callback set 2024-03-25T18:24:44.448 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.445+0000 7fc7917fa640 1 -- 172.21.15.155:0/2107260321 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 2696932296 0 0) 0x7fc780006320 con 0x7fc77800e360 2024-03-25T18:24:44.448 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.445+0000 7fc7917fa640 10 client.4113.objecter ms_dispatch 0x7fc794000b90 osd_map(1..1 src has 1..1) v4 2024-03-25T18:24:44.448 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.445+0000 7fc7917fa640 3 client.4113.objecter handle_osd_map got epochs [1,1] > 0 2024-03-25T18:24:44.448 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.445+0000 7fc7917fa640 3 client.4113.objecter handle_osd_map decoding full epoch 1 2024-03-25T18:24:44.448 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.445+0000 7fc7917fa640 20 client.4113.objecter dump_active .. 0 homeless 2024-03-25T18:24:44.448 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.445+0000 7fc7917fa640 1 -- 172.21.15.155:0/2107260321 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fc780006720 con 0x7fc77800e360 2024-03-25T18:24:44.448 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.445+0000 7fc7917fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:44.448 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.445+0000 7fc7917fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:44.527 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.527+0000 7fc79ab6d640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:24:44.528 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.527+0000 7fc79ab6d640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:24:44.528 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.527+0000 7fc79ab6d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:24:44.528 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.527+0000 7fc79ab6d640 1 -- 172.21.15.155:0/2107260321 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fc7940b14c0 con 0x7fc77800e360 2024-03-25T18:24:44.713 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.712+0000 7fc7917fa640 1 -- 172.21.15.155:0/2107260321 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7fc780005260 con 0x7fc77800e360 2024-03-25T18:24:44.713 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.713+0000 7fc7917fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:24:44.714 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.713+0000 7fc7917fa640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-25T18:24:44.714 INFO:teuthology.orchestra.run.smithi155.stderr:adjusted tunables profile to default 2024-03-25T18:24:44.715 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.713+0000 7fc79ab6d640 10 librados: watch_flush enter 2024-03-25T18:24:44.715 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.713+0000 7fc79ab6d640 10 librados: watch_flush exit 2024-03-25T18:24:44.715 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.713+0000 7fc79ab6d640 20 client.4113.objecter shutdown clearing up homeless session... 2024-03-25T18:24:44.715 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.713+0000 7fc79ab6d640 10 client.4113.objecter successfully canceled tick 2024-03-25T18:24:44.715 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.713+0000 7fc79ab6d640 10 monclient: shutdown 2024-03-25T18:24:44.715 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.713+0000 7fc79ab6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:44.715 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.713+0000 7fc79ab6d640 1 -- 172.21.15.155:0/2107260321 >> v1:172.21.15.155:6789/0 conn(0x7fc77800e360 legacy=0x7fc7940ac750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:44.715 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.714+0000 7fc79ab6d640 1 -- 172.21.15.155:0/2107260321 shutdown_connections 2024-03-25T18:24:44.715 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.714+0000 7fc79ab6d640 1 -- 172.21.15.155:0/2107260321 >> 172.21.15.155:0/2107260321 conn(0x7fc7940b7f40 msgr2=0x7fc7940b3780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:44.716 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.714+0000 7fc79ab6d640 1 -- 172.21.15.155:0/2107260321 shutdown_connections 2024-03-25T18:24:44.716 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.714+0000 7fc79ab6d640 1 -- 172.21.15.155:0/2107260321 wait complete. 2024-03-25T18:24:44.716 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:44.714+0000 7fc79ab6d640 1 librados: shutdown 2024-03-25T18:24:44.726 INFO:tasks.ceph:check_enable_crimson: False 2024-03-25T18:24:44.726 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-25T18:24:44.727 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-25T18:24:44.727 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-25T18:24:44.754 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-25T18:24:44.754 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-25T18:24:44.771 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-25T18:24:44.770+0000 7f91e582f040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T18:24:44.797 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-25T18:24:44.798 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-25T18:24:44.830 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-25T18:24:44.830+0000 7f91e582f040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T18:24:44.857 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-25T18:24:44.857 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-25T18:24:44.861 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-03-25T18:24:44.860+0000 7f1820e2b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T18:24:44.915 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-03-25T18:24:44.914+0000 7f1820e2b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T18:24:44.924 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-03-25T18:24:44.924 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-25T18:24:44.953 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-03-25T18:24:44.953 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-25T18:24:45.021 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-03-25T18:24:45.021 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-25T18:24:45.088 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-03-25T18:24:45.088 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-25T18:24:45.155 DEBUG:teuthology.orchestra.run.smithi167:> sudo ceph --cluster ceph osd new 359f8cfa-5c58-4b11-ad25-4fac6f4d43ba 0 2024-03-25T18:24:45.295 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.294+0000 7fb20bb82640 10 monclient: get_monmap_and_config 2024-03-25T18:24:45.295 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.294+0000 7fb20bb82640 10 monclient: build_initial_monmap 2024-03-25T18:24:45.295 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.294+0000 7fb20bb82640 10 monclient: monmap: 2024-03-25T18:24:45.295 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 0 2024-03-25T18:24:45.295 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:45.295 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-03-25T18:24:45.295399+0000 2024-03-25T18:24:45.295 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-03-25T18:24:45.295399+0000 2024-03-25T18:24:45.296 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:45.296 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 3 2024-03-25T18:24:45.296 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:24:45.296 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:24:45.296 INFO:teuthology.orchestra.run.smithi167.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:24:45.296 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:24:45.297 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.295+0000 7fb20bb82640 1 Processor -- start 2024-03-25T18:24:45.297 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.295+0000 7fb20bb82640 1 -- start start 2024-03-25T18:24:45.297 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.295+0000 7fb20bb82640 10 monclient: init 2024-03-25T18:24:45.297 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.295+0000 7fb20bb82640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:45.297 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.295+0000 7fb20bb82640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:24:45.297 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.295+0000 7fb20bb82640 10 monclient(hunting): picked mon.noname-a con 0x7fb204070210 addr v1:172.21.15.155:6789/0 2024-03-25T18:24:45.297 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.295+0000 7fb20bb82640 10 monclient(hunting): picked mon.noname-b con 0x7fb2040b06d0 addr v1:172.21.15.155:6790/0 2024-03-25T18:24:45.297 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.295+0000 7fb20bb82640 10 monclient(hunting): picked mon.noname-c con 0x7fb2040705f0 addr v1:172.21.15.167:6789/0 2024-03-25T18:24:45.298 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.295+0000 7fb20bb82640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2040b0b80 con 0x7fb204070210 2024-03-25T18:24:45.298 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.295+0000 7fb20bb82640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2040b0430 con 0x7fb2040705f0 2024-03-25T18:24:45.298 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.295+0000 7fb20bb82640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb204071670 con 0x7fb2040b06d0 2024-03-25T18:24:45.298 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.295+0000 7fb20bb82640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:45.298 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.295+0000 7fb20bb82640 10 monclient(hunting): authenticate will time out at 7515.417689s 2024-03-25T18:24:45.298 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.296+0000 7fb20911d640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fb2040b06d0 0x7fb2040afd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.167:51910/0 (socket says 172.21.15.167:51910) 2024-03-25T18:24:45.298 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.296+0000 7fb20911d640 1 -- 172.21.15.167:0/970725931 learned_addr learned my addr 172.21.15.167:0/970725931 (peer_addr_for_me v1:172.21.15.167:0/0) 2024-03-25T18:24:45.299 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.298+0000 7fb20891c640 1 -- 172.21.15.167:0/970725931 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4183895119 0 0) 0x7fb1fc001490 con 0x7fb2040705f0 2024-03-25T18:24:45.299 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.298+0000 7fb20891c640 10 monclient(hunting): _negotiate 2024-03-25T18:24:45.299 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.298+0000 7fb20891c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:45.300 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.298+0000 7fb20891c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:45.300 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.298+0000 7fb20891c640 10 monclient(hunting): my global_id is 4108 2024-03-25T18:24:45.300 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.298+0000 7fb20891c640 1 -- 172.21.15.167:0/970725931 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1f0003520 con 0x7fb2040705f0 2024-03-25T18:24:45.300 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.298+0000 7fb20891c640 1 -- 172.21.15.167:0/970725931 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2486377324 0 0) 0x7fb204071670 con 0x7fb2040b06d0 2024-03-25T18:24:45.300 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.298+0000 7fb20891c640 10 monclient(hunting): _negotiate 2024-03-25T18:24:45.300 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.298+0000 7fb20891c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:45.300 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.298+0000 7fb20891c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:45.300 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.298+0000 7fb20891c640 10 monclient(hunting): my global_id is 4115 2024-03-25T18:24:45.300 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.298+0000 7fb20891c640 1 -- 172.21.15.167:0/970725931 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1fc001490 con 0x7fb2040b06d0 2024-03-25T18:24:45.301 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.299+0000 7fb20891c640 1 -- 172.21.15.167:0/970725931 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 584863731 0 0) 0x7fb2040b0b80 con 0x7fb204070210 2024-03-25T18:24:45.301 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.299+0000 7fb20891c640 10 monclient(hunting): _negotiate 2024-03-25T18:24:45.301 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.299+0000 7fb20891c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:45.301 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.299+0000 7fb20891c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:45.301 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.299+0000 7fb20891c640 10 monclient(hunting): my global_id is 4116 2024-03-25T18:24:45.301 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.299+0000 7fb20891c640 1 -- 172.21.15.167:0/970725931 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb204071670 con 0x7fb204070210 2024-03-25T18:24:45.304 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.303+0000 7fb20891c640 1 -- 172.21.15.167:0/970725931 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2077833021 0 0) 0x7fb204071670 con 0x7fb204070210 2024-03-25T18:24:45.304 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.303+0000 7fb20891c640 1 -- 172.21.15.167:0/970725931 >> v1:172.21.15.155:6790/0 conn(0x7fb2040b06d0 legacy=0x7fb2040afd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:45.304 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.303+0000 7fb20891c640 1 -- 172.21.15.167:0/970725931 >> v1:172.21.15.167:6789/0 conn(0x7fb2040705f0 legacy=0x7fb20414b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:45.304 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.303+0000 7fb20891c640 10 monclient: _finish_hunting 0 2024-03-25T18:24:45.304 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.303+0000 7fb20891c640 1 monclient: found mon.noname-a 2024-03-25T18:24:45.304 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.303+0000 7fb20891c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:45.304 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.303+0000 7fb20891c640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-25T18:24:45.304 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.303+0000 7fb20891c640 1 -- 172.21.15.167:0/970725931 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb204070a60 con 0x7fb204070210 2024-03-25T18:24:45.305 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.303+0000 7fb20891c640 10 monclient: _finish_auth 0 2024-03-25T18:24:45.305 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.303+0000 7fb20891c640 10 monclient: _check_auth_tickets 2024-03-25T18:24:45.305 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.303+0000 7fb20891c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:45.305 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.303+0000 7fb20891c640 1 -- 172.21.15.167:0/970725931 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fb1f40015b0 con 0x7fb204070210 2024-03-25T18:24:45.305 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.303+0000 7fb20891c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:45.305 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.303+0000 7fb20891c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:24:45.305 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.303+0000 7fb20891c640 10 monclient: dump: 2024-03-25T18:24:45.305 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-03-25T18:24:45.305 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:45.305 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:45.305 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:45.305 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:45.305 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 3 2024-03-25T18:24:45.305 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:45.305 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:45.305 INFO:teuthology.orchestra.run.smithi167.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:45.305 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:24:45.306 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.303+0000 7fb20bb82640 5 monclient: authenticate success, global_id 4116 2024-03-25T18:24:45.306 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.303+0000 7fb20bb82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:45.307 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.305+0000 7fb20891c640 1 -- 172.21.15.167:0/970725931 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb1f4001710 con 0x7fb204070210 2024-03-25T18:24:45.307 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.305+0000 7fb20891c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:45.307 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.305+0000 7fb20891c640 1 -- 172.21.15.167:0/970725931 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fb1f4001b60 con 0x7fb204070210 2024-03-25T18:24:45.307 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.305+0000 7fb20891c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:45.307 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.305+0000 7fb20891c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:45.307 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.305+0000 7fb20891c640 10 monclient: dump: 2024-03-25T18:24:45.307 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-03-25T18:24:45.307 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:45.307 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:45.307 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:45.307 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:45.307 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 3 2024-03-25T18:24:45.308 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:45.308 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:45.308 INFO:teuthology.orchestra.run.smithi167.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:45.308 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:24:45.308 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.305+0000 7fb20bb82640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:45.308 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.305+0000 7fb20bb82640 4 set_mon_vals no callback set 2024-03-25T18:24:45.308 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.306+0000 7fb20bb82640 10 monclient: shutdown 2024-03-25T18:24:45.308 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.306+0000 7fb20bb82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:45.308 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.306+0000 7fb20bb82640 1 -- 172.21.15.167:0/970725931 >> v1:172.21.15.155:6789/0 conn(0x7fb204070210 legacy=0x7fb204140930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:45.308 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.306+0000 7fb20bb82640 1 -- 172.21.15.167:0/970725931 shutdown_connections 2024-03-25T18:24:45.308 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.306+0000 7fb20bb82640 1 -- 172.21.15.167:0/970725931 >> 172.21.15.167:0/970725931 conn(0x7fb2040b5cd0 msgr2=0x7fb2040b60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:45.308 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.306+0000 7fb20bb82640 1 -- 172.21.15.167:0/970725931 shutdown_connections 2024-03-25T18:24:45.308 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.306+0000 7fb20bb82640 1 -- 172.21.15.167:0/970725931 wait complete. 2024-03-25T18:24:45.310 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.307+0000 7fb20bb82640 10 monclient: build_initial_monmap 2024-03-25T18:24:45.310 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.307+0000 7fb20bb82640 10 monclient: monmap: 2024-03-25T18:24:45.310 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 0 2024-03-25T18:24:45.310 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:45.310 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 0.000000 2024-03-25T18:24:45.310 INFO:teuthology.orchestra.run.smithi167.stderr:created 0.000000 2024-03-25T18:24:45.311 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:45.311 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 1 2024-03-25T18:24:45.311 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:24:45.311 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:24:45.311 INFO:teuthology.orchestra.run.smithi167.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:24:45.311 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:24:45.311 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.307+0000 7fb20bb82640 1 librados: starting msgr at 2024-03-25T18:24:45.311 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.307+0000 7fb20bb82640 1 librados: starting objecter 2024-03-25T18:24:45.311 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.307+0000 7fb20bb82640 1 Processor -- start 2024-03-25T18:24:45.311 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.307+0000 7fb20bb82640 1 -- start start 2024-03-25T18:24:45.311 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.307+0000 7fb20bb82640 1 librados: setting wanted keys 2024-03-25T18:24:45.311 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.307+0000 7fb20bb82640 1 librados: calling monclient init 2024-03-25T18:24:45.311 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.307+0000 7fb20bb82640 10 monclient: init 2024-03-25T18:24:45.311 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.308+0000 7fb20bb82640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:45.311 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.308+0000 7fb20bb82640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:24:45.312 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.308+0000 7fb20bb82640 10 monclient(hunting): picked mon.noname-a con 0x7fb204070210 addr v1:172.21.15.155:6789/0 2024-03-25T18:24:45.312 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.308+0000 7fb20bb82640 10 monclient(hunting): picked mon.noname-c con 0x7fb2040705f0 addr v1:172.21.15.155:6790/0 2024-03-25T18:24:45.312 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.308+0000 7fb20bb82640 10 monclient(hunting): picked mon.noname-b con 0x7fb2040b06d0 addr v1:172.21.15.167:6789/0 2024-03-25T18:24:45.312 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.308+0000 7fb20bb82640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb20412d7c0 con 0x7fb204070210 2024-03-25T18:24:45.312 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.308+0000 7fb20bb82640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb20406d2d0 con 0x7fb2040b06d0 2024-03-25T18:24:45.312 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.308+0000 7fb20bb82640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb204138860 con 0x7fb2040705f0 2024-03-25T18:24:45.312 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.308+0000 7fb20bb82640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:45.312 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.308+0000 7fb20bb82640 10 monclient(hunting): authenticate will time out at 7515.430596s 2024-03-25T18:24:45.312 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.309+0000 7fb20991e640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fb204070210 0x7fb2041371d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.167:38450/0 (socket says 172.21.15.167:38450) 2024-03-25T18:24:45.312 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.309+0000 7fb20991e640 1 -- 172.21.15.167:0/4035958892 learned_addr learned my addr 172.21.15.167:0/4035958892 (peer_addr_for_me v1:172.21.15.167:0/0) 2024-03-25T18:24:45.312 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.309+0000 7fb1fa7fc640 10 client.?.objecter ms_handle_connect 0x7fb204070210 2024-03-25T18:24:45.312 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.309+0000 7fb1fa7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:45.313 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.312+0000 7fb1fa7fc640 1 -- 172.21.15.167:0/4035958892 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1007612656 0 0) 0x7fb20412d7c0 con 0x7fb204070210 2024-03-25T18:24:45.313 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.312+0000 7fb1fa7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:45.313 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.312+0000 7fb1fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:45.313 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.312+0000 7fb1fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:45.313 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.312+0000 7fb1fa7fc640 10 monclient(hunting): my global_id is 4119 2024-03-25T18:24:45.313 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.312+0000 7fb1fa7fc640 1 -- 172.21.15.167:0/4035958892 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1e8004eb0 con 0x7fb204070210 2024-03-25T18:24:45.318 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.317+0000 7fb1fa7fc640 1 -- 172.21.15.167:0/4035958892 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1188126343 0 0) 0x7fb1e8004eb0 con 0x7fb204070210 2024-03-25T18:24:45.318 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.317+0000 7fb1fa7fc640 1 -- 172.21.15.167:0/4035958892 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb20412d7c0 con 0x7fb204070210 2024-03-25T18:24:45.318 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.317+0000 7fb1fa7fc640 1 -- 172.21.15.167:0/4035958892 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fb1f4001ae0 con 0x7fb204070210 2024-03-25T18:24:45.318 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.317+0000 7fb1fa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:45.318 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.317+0000 7fb1fa7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:24:45.318 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.317+0000 7fb1fa7fc640 10 monclient(hunting): dump: 2024-03-25T18:24:45.319 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-03-25T18:24:45.319 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:45.319 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:45.319 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:45.319 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:45.319 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 3 2024-03-25T18:24:45.319 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:45.319 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:45.319 INFO:teuthology.orchestra.run.smithi167.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:45.319 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:24:45.319 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.317+0000 7fb1fa7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:45.323 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.321+0000 7fb1fa7fc640 1 -- 172.21.15.167:0/4035958892 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1404568853 0 0) 0x7fb20412d7c0 con 0x7fb204070210 2024-03-25T18:24:45.323 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.321+0000 7fb1fa7fc640 1 -- 172.21.15.167:0/4035958892 >> v1:172.21.15.155:6790/0 conn(0x7fb2040705f0 legacy=0x7fb20412c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:45.323 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.321+0000 7fb1fa7fc640 1 -- 172.21.15.167:0/4035958892 >> v1:172.21.15.167:6789/0 conn(0x7fb2040b06d0 legacy=0x7fb20412d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:45.323 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.321+0000 7fb1fa7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:24:45.323 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.321+0000 7fb1fa7fc640 1 monclient: found mon.a 2024-03-25T18:24:45.323 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.321+0000 7fb1fa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:45.323 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.321+0000 7fb1fa7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:24:45.323 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.321+0000 7fb1fa7fc640 1 -- 172.21.15.167:0/4035958892 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb204138b40 con 0x7fb204070210 2024-03-25T18:24:45.323 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.321+0000 7fb1fa7fc640 10 monclient: _finish_auth 0 2024-03-25T18:24:45.323 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.321+0000 7fb1fa7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:24:45.323 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.321+0000 7fb1fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:45.323 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.321+0000 7fb20bb82640 5 monclient: authenticate success, global_id 4119 2024-03-25T18:24:45.323 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.321+0000 7fb20bb82640 10 monclient: _renew_subs 2024-03-25T18:24:45.324 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.321+0000 7fb20bb82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:24:45.324 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.321+0000 7fb20bb82640 1 -- 172.21.15.167:0/4035958892 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2041378e0 con 0x7fb204070210 2024-03-25T18:24:45.324 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.321+0000 7fb20bb82640 10 client.4119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:45.324 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.321+0000 7fb20bb82640 10 monclient: _renew_subs 2024-03-25T18:24:45.324 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.321+0000 7fb20bb82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:24:45.324 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.321+0000 7fb20bb82640 1 -- 172.21.15.167:0/4035958892 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb204139140 con 0x7fb204070210 2024-03-25T18:24:45.324 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.321+0000 7fb20bb82640 1 librados: init done 2024-03-25T18:24:45.324 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.322+0000 7fb20bb82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:45.324 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.322+0000 7fb20bb82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:45.324 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.322+0000 7fb20bb82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:24:45.324 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.322+0000 7fb20bb82640 1 -- 172.21.15.167:0/4035958892 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb20404f730 con 0x7fb204070210 2024-03-25T18:24:45.421 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.420+0000 7fb1fa7fc640 1 -- 172.21.15.167:0/4035958892 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb1f40038a0 con 0x7fb204070210 2024-03-25T18:24:45.421 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.420+0000 7fb1fa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:45.422 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.420+0000 7fb1fa7fc640 1 -- 172.21.15.167:0/4035958892 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fb1f4003c70 con 0x7fb204070210 2024-03-25T18:24:45.422 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.420+0000 7fb1fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:45.422 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.420+0000 7fb1fa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:45.422 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.420+0000 7fb1fa7fc640 10 monclient: dump: 2024-03-25T18:24:45.422 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-03-25T18:24:45.422 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:45.422 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:45.422 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:45.422 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:45.422 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 3 2024-03-25T18:24:45.422 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:45.422 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:45.422 INFO:teuthology.orchestra.run.smithi167.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:45.422 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:24:45.422 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.420+0000 7fb1fa7fc640 1 -- 172.21.15.167:0/4035958892 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fb1f4004220 con 0x7fb204070210 2024-03-25T18:24:45.423 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.420+0000 7fb1fbfff640 4 set_mon_vals no callback set 2024-03-25T18:24:45.423 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.421+0000 7fb1fa7fc640 1 -- 172.21.15.167:0/4035958892 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 3239816286 0 0) 0x7fb1f4006830 con 0x7fb204070210 2024-03-25T18:24:45.423 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.421+0000 7fb1fa7fc640 10 client.4119.objecter ms_dispatch 0x7fb204000b90 osd_map(2..2 src has 1..2) v4 2024-03-25T18:24:45.423 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.421+0000 7fb1fa7fc640 3 client.4119.objecter handle_osd_map got epochs [2,2] > 0 2024-03-25T18:24:45.423 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.421+0000 7fb1fa7fc640 3 client.4119.objecter handle_osd_map decoding full epoch 2 2024-03-25T18:24:45.423 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.421+0000 7fb1fa7fc640 20 client.4119.objecter dump_active .. 0 homeless 2024-03-25T18:24:45.424 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.421+0000 7fb1fa7fc640 1 -- 172.21.15.167:0/4035958892 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fb1f4006a50 con 0x7fb204070210 2024-03-25T18:24:45.424 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.421+0000 7fb1fa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:45.424 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.421+0000 7fb1fa7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:45.428 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-25T18:24:45.427+0000 7f91e582f040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T18:24:45.481 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-25T18:24:45.481+0000 7f91e582f040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T18:24:45.485 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-03-25T18:24:45.484+0000 7f1820e2b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T18:24:45.493 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.492+0000 7fb20bb82640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "359f8cfa-5c58-4b11-ad25-4fac6f4d43ba", "id": 0}] 2024-03-25T18:24:45.493 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.492+0000 7fb20bb82640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "359f8cfa-5c58-4b11-ad25-4fac6f4d43ba", "id": 0}] 2024-03-25T18:24:45.493 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.492+0000 7fb20bb82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:24:45.493 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.492+0000 7fb20bb82640 1 -- 172.21.15.167:0/4035958892 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd new", "uuid": "359f8cfa-5c58-4b11-ad25-4fac6f4d43ba", "id": 0} v 0) v1 -- 0x7fb20413ff90 con 0x7fb204070210 2024-03-25T18:24:45.527 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-25T18:24:45.526+0000 7f91e582f040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T18:24:45.538 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-03-25T18:24:45.537+0000 7f1820e2b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T18:24:45.574 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.574+0000 7fb1fa7fc640 1 -- 172.21.15.167:0/4035958892 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "359f8cfa-5c58-4b11-ad25-4fac6f4d43ba", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 1090709944 0 732707515) 0x7fb1f4005850 con 0x7fb204070210 2024-03-25T18:24:45.575 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.574+0000 7fb1fa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "359f8cfa-5c58-4b11-ad25-4fac6f4d43ba", "id": 0}] 2024-03-25T18:24:45.575 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.574+0000 7fb1fa7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:45.575 INFO:teuthology.orchestra.run.smithi167.stdout:0 2024-03-25T18:24:45.576 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.574+0000 7fb20bb82640 10 librados: watch_flush enter 2024-03-25T18:24:45.576 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.575+0000 7fb20bb82640 10 librados: watch_flush exit 2024-03-25T18:24:45.576 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.575+0000 7fb20bb82640 20 client.4119.objecter shutdown clearing up homeless session... 2024-03-25T18:24:45.576 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.575+0000 7fb20bb82640 10 client.4119.objecter successfully canceled tick 2024-03-25T18:24:45.577 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.575+0000 7fb20bb82640 10 monclient: shutdown 2024-03-25T18:24:45.577 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.575+0000 7fb20bb82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:45.577 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.575+0000 7fb20bb82640 1 -- 172.21.15.167:0/4035958892 >> v1:172.21.15.155:6789/0 conn(0x7fb204070210 legacy=0x7fb2041371d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:45.577 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.575+0000 7fb20bb82640 1 -- 172.21.15.167:0/4035958892 shutdown_connections 2024-03-25T18:24:45.577 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.575+0000 7fb20bb82640 1 -- 172.21.15.167:0/4035958892 >> 172.21.15.167:0/4035958892 conn(0x7fb2040b5cd0 msgr2=0x7fb204074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:45.577 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.575+0000 7fb20bb82640 1 -- 172.21.15.167:0/4035958892 shutdown_connections 2024-03-25T18:24:45.577 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.575+0000 7fb20bb82640 1 -- 172.21.15.167:0/4035958892 wait complete. 2024-03-25T18:24:45.577 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.575+0000 7fb20bb82640 1 librados: shutdown 2024-03-25T18:24:45.584 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-03-25T18:24:45.583+0000 7f1820e2b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T18:24:45.586 DEBUG:teuthology.orchestra.run.smithi167:> sudo ceph --cluster ceph osd new d22b6af2-efd4-4433-a24d-1117b2c8681d 1 2024-03-25T18:24:45.593 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-25T18:24:45.593+0000 7f91e582f040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T18:24:45.642 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-03-25T18:24:45.642+0000 7f1820e2b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T18:24:45.643 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-25T18:24:45.643+0000 7f91e582f040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T18:24:45.686 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.684+0000 7f205c6f1640 10 monclient: get_monmap_and_config 2024-03-25T18:24:45.686 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.684+0000 7f205c6f1640 10 monclient: build_initial_monmap 2024-03-25T18:24:45.686 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.684+0000 7f205c6f1640 10 monclient: monmap: 2024-03-25T18:24:45.686 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 0 2024-03-25T18:24:45.686 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:45.686 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-03-25T18:24:45.685828+0000 2024-03-25T18:24:45.686 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-03-25T18:24:45.685828+0000 2024-03-25T18:24:45.686 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:45.687 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 3 2024-03-25T18:24:45.687 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:24:45.687 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:24:45.687 INFO:teuthology.orchestra.run.smithi167.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:24:45.687 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:24:45.687 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.685+0000 7f205c6f1640 1 Processor -- start 2024-03-25T18:24:45.687 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.685+0000 7f205c6f1640 1 -- start start 2024-03-25T18:24:45.687 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.685+0000 7f205c6f1640 10 monclient: init 2024-03-25T18:24:45.687 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-03-25T18:24:45.685+0000 7f1820e2b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T18:24:45.688 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.686+0000 7f205c6f1640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:45.688 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.686+0000 7f205c6f1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:24:45.689 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.686+0000 7f205c6f1640 10 monclient(hunting): picked mon.noname-b con 0x7f20540b4700 addr v1:172.21.15.155:6790/0 2024-03-25T18:24:45.689 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.686+0000 7f205c6f1640 10 monclient(hunting): picked mon.noname-a con 0x7f2054142a20 addr v1:172.21.15.155:6789/0 2024-03-25T18:24:45.689 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.686+0000 7f205c6f1640 10 monclient(hunting): picked mon.noname-c con 0x7f20540a96b0 addr v1:172.21.15.167:6789/0 2024-03-25T18:24:45.689 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.686+0000 7f205c6f1640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20540a9d60 con 0x7f2054142a20 2024-03-25T18:24:45.689 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.686+0000 7f205c6f1640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f205413ff10 con 0x7f20540a96b0 2024-03-25T18:24:45.689 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.686+0000 7f205c6f1640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20540aa1f0 con 0x7f20540b4700 2024-03-25T18:24:45.689 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.686+0000 7f205c6f1640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:45.689 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.686+0000 7f205c6f1640 10 monclient(hunting): authenticate will time out at 7515.808164s 2024-03-25T18:24:45.689 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.686+0000 7f205a48d640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f20540b4700 0x7f2054142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.167:51918/0 (socket says 172.21.15.167:51918) 2024-03-25T18:24:45.689 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.687+0000 7f205a48d640 1 -- 172.21.15.167:0/2099797820 learned_addr learned my addr 172.21.15.167:0/2099797820 (peer_addr_for_me v1:172.21.15.167:0/0) 2024-03-25T18:24:45.691 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.689+0000 7f205948b640 1 -- 172.21.15.167:0/2099797820 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2827194756 0 0) 0x7f20540a9d60 con 0x7f2054142a20 2024-03-25T18:24:45.691 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.689+0000 7f205948b640 10 monclient(hunting): _negotiate 2024-03-25T18:24:45.691 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.689+0000 7f205948b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:45.691 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.689+0000 7f205948b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:45.691 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.689+0000 7f205948b640 10 monclient(hunting): my global_id is 4122 2024-03-25T18:24:45.691 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.689+0000 7f205948b640 1 -- 172.21.15.167:0/2099797820 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2040003520 con 0x7f2054142a20 2024-03-25T18:24:45.691 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.689+0000 7f205948b640 1 -- 172.21.15.167:0/2099797820 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3289436208 0 0) 0x7f20540aa1f0 con 0x7f20540b4700 2024-03-25T18:24:45.691 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.689+0000 7f205948b640 10 monclient(hunting): _negotiate 2024-03-25T18:24:45.691 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.689+0000 7f205948b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:45.691 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.689+0000 7f205948b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:45.691 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.689+0000 7f205948b640 10 monclient(hunting): my global_id is 4118 2024-03-25T18:24:45.691 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.689+0000 7f205948b640 1 -- 172.21.15.167:0/2099797820 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20540a9d60 con 0x7f20540b4700 2024-03-25T18:24:45.692 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.689+0000 7f205948b640 1 -- 172.21.15.167:0/2099797820 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2375405965 0 0) 0x7f205413ff10 con 0x7f20540a96b0 2024-03-25T18:24:45.692 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.689+0000 7f205948b640 10 monclient(hunting): _negotiate 2024-03-25T18:24:45.692 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.689+0000 7f205948b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:45.692 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.689+0000 7f205948b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:45.692 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.689+0000 7f205948b640 10 monclient(hunting): my global_id is 4111 2024-03-25T18:24:45.692 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.689+0000 7f205948b640 1 -- 172.21.15.167:0/2099797820 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20540aa1f0 con 0x7f20540a96b0 2024-03-25T18:24:45.693 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.692+0000 7f205948b640 1 -- 172.21.15.167:0/2099797820 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2781663438 0 0) 0x7f2040003520 con 0x7f2054142a20 2024-03-25T18:24:45.693 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.692+0000 7f205948b640 1 -- 172.21.15.167:0/2099797820 >> v1:172.21.15.155:6790/0 conn(0x7f20540b4700 legacy=0x7f2054142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:45.694 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.692+0000 7f205948b640 1 -- 172.21.15.167:0/2099797820 >> v1:172.21.15.167:6789/0 conn(0x7f20540a96b0 legacy=0x7f205413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:45.694 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.692+0000 7f205948b640 10 monclient: _finish_hunting 0 2024-03-25T18:24:45.694 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.692+0000 7f205948b640 1 monclient: found mon.noname-a 2024-03-25T18:24:45.694 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.692+0000 7f205948b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:45.694 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.692+0000 7f205948b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-25T18:24:45.694 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.692+0000 7f205948b640 1 -- 172.21.15.167:0/2099797820 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20540b4ae0 con 0x7f2054142a20 2024-03-25T18:24:45.694 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.692+0000 7f205948b640 10 monclient: _finish_auth 0 2024-03-25T18:24:45.694 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.692+0000 7f205948b640 10 monclient: _check_auth_tickets 2024-03-25T18:24:45.694 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.692+0000 7f205948b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:45.694 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.692+0000 7f205948b640 1 -- 172.21.15.167:0/2099797820 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f2050001690 con 0x7f2054142a20 2024-03-25T18:24:45.694 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.692+0000 7f205948b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:45.694 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.692+0000 7f205948b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:24:45.694 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.692+0000 7f205948b640 10 monclient: dump: 2024-03-25T18:24:45.694 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-03-25T18:24:45.694 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:45.695 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:45.695 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:45.695 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:45.695 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 3 2024-03-25T18:24:45.695 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:45.695 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:45.695 INFO:teuthology.orchestra.run.smithi167.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:45.695 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:24:45.695 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.692+0000 7f205c6f1640 5 monclient: authenticate success, global_id 4122 2024-03-25T18:24:45.695 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.692+0000 7f205c6f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:45.697 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.694+0000 7f205948b640 1 -- 172.21.15.167:0/2099797820 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f20500017f0 con 0x7f2054142a20 2024-03-25T18:24:45.697 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.694+0000 7f205948b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:45.697 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.694+0000 7f205948b640 1 -- 172.21.15.167:0/2099797820 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f2050001c00 con 0x7f2054142a20 2024-03-25T18:24:45.697 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.694+0000 7f205c6f1640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:45.697 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.694+0000 7f205c6f1640 4 set_mon_vals no callback set 2024-03-25T18:24:45.697 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.694+0000 7f205948b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:45.697 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.694+0000 7f205948b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:45.697 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.694+0000 7f205948b640 10 monclient: dump: 2024-03-25T18:24:45.697 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-03-25T18:24:45.697 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:45.697 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:45.697 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:45.697 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:45.697 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 3 2024-03-25T18:24:45.697 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:45.698 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:45.698 INFO:teuthology.orchestra.run.smithi167.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:45.698 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:24:45.698 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.695+0000 7f205c6f1640 10 monclient: shutdown 2024-03-25T18:24:45.698 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.695+0000 7f205c6f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:45.698 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.695+0000 7f205c6f1640 1 -- 172.21.15.167:0/2099797820 >> v1:172.21.15.155:6789/0 conn(0x7f2054142a20 legacy=0x7f205413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:45.698 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.695+0000 7f205c6f1640 1 -- 172.21.15.167:0/2099797820 shutdown_connections 2024-03-25T18:24:45.698 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.695+0000 7f205c6f1640 1 -- 172.21.15.167:0/2099797820 >> 172.21.15.167:0/2099797820 conn(0x7f20540668e0 msgr2=0x7f2054066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:45.698 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.695+0000 7f205c6f1640 1 -- 172.21.15.167:0/2099797820 shutdown_connections 2024-03-25T18:24:45.698 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.695+0000 7f205c6f1640 1 -- 172.21.15.167:0/2099797820 wait complete. 2024-03-25T18:24:45.698 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.696+0000 7f205c6f1640 10 monclient: build_initial_monmap 2024-03-25T18:24:45.698 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.696+0000 7f205c6f1640 10 monclient: monmap: 2024-03-25T18:24:45.698 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 0 2024-03-25T18:24:45.698 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:45.698 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 0.000000 2024-03-25T18:24:45.698 INFO:teuthology.orchestra.run.smithi167.stderr:created 0.000000 2024-03-25T18:24:45.699 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:45.699 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 1 2024-03-25T18:24:45.699 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:24:45.699 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:24:45.699 INFO:teuthology.orchestra.run.smithi167.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:24:45.699 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:24:45.700 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.696+0000 7f205c6f1640 1 librados: starting msgr at 2024-03-25T18:24:45.701 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.696+0000 7f205c6f1640 1 librados: starting objecter 2024-03-25T18:24:45.701 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.696+0000 7f205c6f1640 1 Processor -- start 2024-03-25T18:24:45.701 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.697+0000 7f205c6f1640 1 -- start start 2024-03-25T18:24:45.701 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.697+0000 7f205c6f1640 1 librados: setting wanted keys 2024-03-25T18:24:45.701 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.697+0000 7f205c6f1640 1 librados: calling monclient init 2024-03-25T18:24:45.701 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.697+0000 7f205c6f1640 10 monclient: init 2024-03-25T18:24:45.701 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.697+0000 7f205c6f1640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:45.701 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.697+0000 7f205c6f1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:24:45.701 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.697+0000 7f205c6f1640 10 monclient(hunting): picked mon.noname-c con 0x7f20540a96b0 addr v1:172.21.15.155:6790/0 2024-03-25T18:24:45.701 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.697+0000 7f205c6f1640 10 monclient(hunting): picked mon.noname-a con 0x7f20540b4700 addr v1:172.21.15.155:6789/0 2024-03-25T18:24:45.701 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.697+0000 7f205c6f1640 10 monclient(hunting): picked mon.noname-b con 0x7f2054142a20 addr v1:172.21.15.167:6789/0 2024-03-25T18:24:45.701 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.697+0000 7f205c6f1640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f205413a0a0 con 0x7f20540b4700 2024-03-25T18:24:45.701 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.697+0000 7f205c6f1640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20541385d0 con 0x7f2054142a20 2024-03-25T18:24:45.701 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.697+0000 7f205c6f1640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f205413a310 con 0x7f20540a96b0 2024-03-25T18:24:45.702 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.697+0000 7f205c6f1640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:45.702 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.697+0000 7f205c6f1640 10 monclient(hunting): authenticate will time out at 7515.819691s 2024-03-25T18:24:45.702 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.698+0000 7f2059c8c640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f20540b4700 0x7f205413add0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.167:38464/0 (socket says 172.21.15.167:38464) 2024-03-25T18:24:45.702 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.698+0000 7f2059c8c640 1 -- 172.21.15.167:0/3848068340 learned_addr learned my addr 172.21.15.167:0/3848068340 (peer_addr_for_me v1:172.21.15.167:0/0) 2024-03-25T18:24:45.702 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.698+0000 7f204affd640 10 client.?.objecter ms_handle_connect 0x7f20540b4700 2024-03-25T18:24:45.702 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.698+0000 7f204affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:45.702 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.700+0000 7f204affd640 10 client.?.objecter ms_handle_connect 0x7f20540a96b0 2024-03-25T18:24:45.702 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.700+0000 7f204affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:45.703 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.700+0000 7f204affd640 1 -- 172.21.15.167:0/3848068340 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1563666202 0 0) 0x7f205413a0a0 con 0x7f20540b4700 2024-03-25T18:24:45.703 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.700+0000 7f204affd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:45.703 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.700+0000 7f204affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:45.703 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.700+0000 7f204affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:45.703 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.700+0000 7f204affd640 10 monclient(hunting): my global_id is 4125 2024-03-25T18:24:45.703 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.700+0000 7f204affd640 1 -- 172.21.15.167:0/3848068340 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2038004eb0 con 0x7f20540b4700 2024-03-25T18:24:45.703 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.701+0000 7f204affd640 10 client.?.objecter ms_handle_connect 0x7f2054142a20 2024-03-25T18:24:45.703 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.701+0000 7f204affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:45.703 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.702+0000 7f204affd640 1 -- 172.21.15.167:0/3848068340 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2567530636 0 0) 0x7f205413a310 con 0x7f20540a96b0 2024-03-25T18:24:45.704 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.702+0000 7f204affd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:45.704 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.702+0000 7f204affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:45.704 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.702+0000 7f204affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:45.704 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.702+0000 7f204affd640 10 monclient(hunting): my global_id is 4121 2024-03-25T18:24:45.704 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.702+0000 7f204affd640 1 -- 172.21.15.167:0/3848068340 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f205413a0a0 con 0x7f20540a96b0 2024-03-25T18:24:45.704 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.703+0000 7f204affd640 1 -- 172.21.15.167:0/3848068340 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2783875860 0 0) 0x7f20541385d0 con 0x7f2054142a20 2024-03-25T18:24:45.704 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.703+0000 7f204affd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:45.704 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.703+0000 7f204affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:45.704 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.703+0000 7f204affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:45.705 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.703+0000 7f204affd640 10 monclient(hunting): my global_id is 4114 2024-03-25T18:24:45.705 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.703+0000 7f204affd640 1 -- 172.21.15.167:0/3848068340 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f205413a310 con 0x7f2054142a20 2024-03-25T18:24:45.706 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.705+0000 7f204affd640 1 -- 172.21.15.167:0/3848068340 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2612431828 0 0) 0x7f2038004eb0 con 0x7f20540b4700 2024-03-25T18:24:45.706 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.705+0000 7f204affd640 1 -- 172.21.15.167:0/3848068340 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f20541385d0 con 0x7f20540b4700 2024-03-25T18:24:45.706 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.705+0000 7f204affd640 1 -- 172.21.15.167:0/3848068340 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f2050001bc0 con 0x7f20540b4700 2024-03-25T18:24:45.706 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.705+0000 7f204affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:45.706 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.705+0000 7f204affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:24:45.706 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.705+0000 7f204affd640 10 monclient(hunting): dump: 2024-03-25T18:24:45.706 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-03-25T18:24:45.706 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:45.707 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:45.707 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:45.707 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:45.707 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 3 2024-03-25T18:24:45.707 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:45.707 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:45.707 INFO:teuthology.orchestra.run.smithi167.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:45.707 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:24:45.707 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.705+0000 7f204affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:45.710 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.708+0000 7f204affd640 1 -- 172.21.15.167:0/3848068340 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3124069888 0 0) 0x7f20541385d0 con 0x7f20540b4700 2024-03-25T18:24:45.710 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.708+0000 7f204affd640 1 -- 172.21.15.167:0/3848068340 >> v1:172.21.15.155:6790/0 conn(0x7f20540a96b0 legacy=0x7f2054138ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:45.710 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.708+0000 7f204affd640 1 -- 172.21.15.167:0/3848068340 >> v1:172.21.15.167:6789/0 conn(0x7f2054142a20 legacy=0x7f20541e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:45.710 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.708+0000 7f204affd640 10 monclient: _finish_hunting 0 2024-03-25T18:24:45.710 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.708+0000 7f204affd640 1 monclient: found mon.a 2024-03-25T18:24:45.710 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.708+0000 7f204affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:45.710 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.708+0000 7f204affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:24:45.710 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.708+0000 7f204affd640 1 -- 172.21.15.167:0/3848068340 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f205413a5f0 con 0x7f20540b4700 2024-03-25T18:24:45.711 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.708+0000 7f204affd640 10 monclient: _finish_auth 0 2024-03-25T18:24:45.711 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.708+0000 7f204affd640 10 monclient: _check_auth_tickets 2024-03-25T18:24:45.711 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.708+0000 7f204affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:45.711 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.708+0000 7f205c6f1640 5 monclient: authenticate success, global_id 4125 2024-03-25T18:24:45.711 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.708+0000 7f205c6f1640 10 monclient: _renew_subs 2024-03-25T18:24:45.711 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.708+0000 7f205c6f1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:24:45.711 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.708+0000 7f205c6f1640 1 -- 172.21.15.167:0/3848068340 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2054142050 con 0x7f20540b4700 2024-03-25T18:24:45.711 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.708+0000 7f205c6f1640 10 client.4125.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:45.711 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.708+0000 7f205c6f1640 10 monclient: _renew_subs 2024-03-25T18:24:45.711 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.708+0000 7f205c6f1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:24:45.711 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.708+0000 7f205c6f1640 1 -- 172.21.15.167:0/3848068340 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2054136940 con 0x7f20540b4700 2024-03-25T18:24:45.711 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.708+0000 7f205c6f1640 1 librados: init done 2024-03-25T18:24:45.712 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.710+0000 7f205c6f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:45.712 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.710+0000 7f205c6f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:45.712 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.710+0000 7f205c6f1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:24:45.712 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.710+0000 7f205c6f1640 1 -- 172.21.15.167:0/3848068340 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f205413ab10 con 0x7f20540b4700 2024-03-25T18:24:45.713 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.712+0000 7f204affd640 1 -- 172.21.15.167:0/3848068340 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2050001210 con 0x7f20540b4700 2024-03-25T18:24:45.714 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.712+0000 7f204affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:45.714 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.712+0000 7f204affd640 1 -- 172.21.15.167:0/3848068340 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f20500038a0 con 0x7f20540b4700 2024-03-25T18:24:45.714 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.712+0000 7f204affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:45.714 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.712+0000 7f204affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:45.714 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.712+0000 7f204affd640 10 monclient: dump: 2024-03-25T18:24:45.714 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-03-25T18:24:45.714 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:45.714 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:45.714 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:45.714 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:45.714 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 3 2024-03-25T18:24:45.714 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:45.714 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:45.714 INFO:teuthology.orchestra.run.smithi167.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:45.714 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:24:45.715 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.712+0000 7f204affd640 1 -- 172.21.15.167:0/3848068340 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f2050003e50 con 0x7f20540b4700 2024-03-25T18:24:45.715 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.712+0000 7f2058c8a640 4 set_mon_vals no callback set 2024-03-25T18:24:45.715 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.712+0000 7f204affd640 1 -- 172.21.15.167:0/3848068340 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 635367867 0 0) 0x7f2050006560 con 0x7f20540b4700 2024-03-25T18:24:45.715 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.712+0000 7f204affd640 10 client.4125.objecter ms_dispatch 0x7f2054000b90 osd_map(3..3 src has 1..3) v4 2024-03-25T18:24:45.715 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.712+0000 7f204affd640 3 client.4125.objecter handle_osd_map got epochs [3,3] > 0 2024-03-25T18:24:45.715 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.712+0000 7f204affd640 3 client.4125.objecter handle_osd_map decoding full epoch 3 2024-03-25T18:24:45.715 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.712+0000 7f204affd640 20 client.4125.objecter dump_active .. 0 homeless 2024-03-25T18:24:45.760 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-25T18:24:45.759+0000 7f91e582f040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T18:24:45.804 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.803+0000 7f204affd640 1 -- 172.21.15.167:0/3848068340 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f2050004100 con 0x7f20540b4700 2024-03-25T18:24:45.804 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.803+0000 7f204affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:45.804 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.803+0000 7f204affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:45.805 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-03-25T18:24:45.805+0000 7f1820e2b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T18:24:45.829 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-25T18:24:45.828+0000 7f91e582f040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T18:24:45.874 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-25T18:24:45.873+0000 7f91e582f040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T18:24:45.875 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.874+0000 7f205c6f1640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d22b6af2-efd4-4433-a24d-1117b2c8681d", "id": 1}] 2024-03-25T18:24:45.875 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.874+0000 7f205c6f1640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d22b6af2-efd4-4433-a24d-1117b2c8681d", "id": 1}] 2024-03-25T18:24:45.875 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.874+0000 7f205c6f1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:24:45.875 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.874+0000 7f205c6f1640 1 -- 172.21.15.167:0/3848068340 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd new", "uuid": "d22b6af2-efd4-4433-a24d-1117b2c8681d", "id": 1} v 0) v1 -- 0x7f205404f730 con 0x7f20540b4700 2024-03-25T18:24:45.876 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-03-25T18:24:45.876+0000 7f1820e2b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T18:24:45.911 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.910+0000 7f204affd640 1 -- 172.21.15.167:0/3848068340 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d22b6af2-efd4-4433-a24d-1117b2c8681d", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 3969126125 0 940483276) 0x7f2050004450 con 0x7f20540b4700 2024-03-25T18:24:45.911 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.910+0000 7f204affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d22b6af2-efd4-4433-a24d-1117b2c8681d", "id": 1}] 2024-03-25T18:24:45.911 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.910+0000 7f204affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:45.911 INFO:teuthology.orchestra.run.smithi167.stdout:1 2024-03-25T18:24:45.912 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.911+0000 7f205c6f1640 10 librados: watch_flush enter 2024-03-25T18:24:45.912 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.911+0000 7f205c6f1640 10 librados: watch_flush exit 2024-03-25T18:24:45.913 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.911+0000 7f205c6f1640 20 client.4125.objecter shutdown clearing up homeless session... 2024-03-25T18:24:45.913 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.911+0000 7f205c6f1640 10 client.4125.objecter successfully canceled tick 2024-03-25T18:24:45.913 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.911+0000 7f205c6f1640 10 monclient: shutdown 2024-03-25T18:24:45.913 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.911+0000 7f205c6f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:45.913 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.911+0000 7f205c6f1640 1 -- 172.21.15.167:0/3848068340 >> v1:172.21.15.155:6789/0 conn(0x7f20540b4700 legacy=0x7f205413add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:45.913 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.911+0000 7f205c6f1640 1 -- 172.21.15.167:0/3848068340 shutdown_connections 2024-03-25T18:24:45.913 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.911+0000 7f205c6f1640 1 -- 172.21.15.167:0/3848068340 >> 172.21.15.167:0/3848068340 conn(0x7f20540668e0 msgr2=0x7f205413f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:45.913 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.911+0000 7f205c6f1640 1 -- 172.21.15.167:0/3848068340 shutdown_connections 2024-03-25T18:24:45.913 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.911+0000 7f205c6f1640 1 -- 172.21.15.167:0/3848068340 wait complete. 2024-03-25T18:24:45.913 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:45.911+0000 7f205c6f1640 1 librados: shutdown 2024-03-25T18:24:45.925 DEBUG:teuthology.orchestra.run.smithi167:> sudo ceph --cluster ceph osd new 03747c34-7e23-428a-8e35-b6132f7e8ae7 2 2024-03-25T18:24:45.925 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-03-25T18:24:45.924+0000 7f1820e2b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T18:24:45.944 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-25T18:24:45.943+0000 7f91e582f040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T18:24:45.984 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-03-25T18:24:45.983+0000 7f1820e2b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T18:24:45.996 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-25T18:24:45.995+0000 7f91e582f040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T18:24:46.025 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.024+0000 7ffad4e82640 10 monclient: get_monmap_and_config 2024-03-25T18:24:46.025 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.024+0000 7ffad4e82640 10 monclient: build_initial_monmap 2024-03-25T18:24:46.025 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.024+0000 7ffad4e82640 10 monclient: monmap: 2024-03-25T18:24:46.026 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 0 2024-03-25T18:24:46.026 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:46.026 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-03-25T18:24:46.025624+0000 2024-03-25T18:24:46.026 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-03-25T18:24:46.025624+0000 2024-03-25T18:24:46.026 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:46.026 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 3 2024-03-25T18:24:46.026 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:24:46.026 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:24:46.026 INFO:teuthology.orchestra.run.smithi167.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:24:46.026 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:24:46.028 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.025+0000 7ffad4e82640 1 Processor -- start 2024-03-25T18:24:46.028 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.025+0000 7ffad4e82640 1 -- start start 2024-03-25T18:24:46.028 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.025+0000 7ffad4e82640 10 monclient: init 2024-03-25T18:24:46.028 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.026+0000 7ffad4e82640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:46.028 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.026+0000 7ffad4e82640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:24:46.028 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.026+0000 7ffad4e82640 10 monclient(hunting): picked mon.noname-b con 0x7ffad00b4700 addr v1:172.21.15.155:6790/0 2024-03-25T18:24:46.028 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.026+0000 7ffad4e82640 10 monclient(hunting): picked mon.noname-a con 0x7ffad0142a20 addr v1:172.21.15.155:6789/0 2024-03-25T18:24:46.028 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.026+0000 7ffad4e82640 10 monclient(hunting): picked mon.noname-c con 0x7ffad00a96b0 addr v1:172.21.15.167:6789/0 2024-03-25T18:24:46.028 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.026+0000 7ffad4e82640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffad00a9d60 con 0x7ffad0142a20 2024-03-25T18:24:46.028 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.026+0000 7ffad4e82640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffad013ff10 con 0x7ffad00a96b0 2024-03-25T18:24:46.028 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.026+0000 7ffad4e82640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffad00aa1f0 con 0x7ffad00b4700 2024-03-25T18:24:46.028 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.026+0000 7ffad4e82640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:46.028 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.026+0000 7ffad4e82640 10 monclient(hunting): authenticate will time out at 7516.147922s 2024-03-25T18:24:46.028 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.026+0000 7fface59c640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7ffad00b4700 0x7ffad0142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.167:51930/0 (socket says 172.21.15.167:51930) 2024-03-25T18:24:46.029 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.026+0000 7fface59c640 1 -- 172.21.15.167:0/4118980630 learned_addr learned my addr 172.21.15.167:0/4118980630 (peer_addr_for_me v1:172.21.15.167:0/0) 2024-03-25T18:24:46.030 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.029+0000 7ffacdd9b640 1 -- 172.21.15.167:0/4118980630 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4171194633 0 0) 0x7ffad00aa1f0 con 0x7ffad00b4700 2024-03-25T18:24:46.030 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.029+0000 7ffacdd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:24:46.030 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.029+0000 7ffacdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:46.030 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.029+0000 7ffacdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:46.030 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.029+0000 7ffacdd9b640 10 monclient(hunting): my global_id is 4124 2024-03-25T18:24:46.030 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.029+0000 7ffacdd9b640 1 -- 172.21.15.167:0/4118980630 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffabc003520 con 0x7ffad00b4700 2024-03-25T18:24:46.031 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.029+0000 7ffacdd9b640 1 -- 172.21.15.167:0/4118980630 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1430705871 0 0) 0x7ffad013ff10 con 0x7ffad00a96b0 2024-03-25T18:24:46.031 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.029+0000 7ffacdd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:24:46.031 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.029+0000 7ffacdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:46.031 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.029+0000 7ffacdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:46.031 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.029+0000 7ffacdd9b640 10 monclient(hunting): my global_id is 4117 2024-03-25T18:24:46.031 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.029+0000 7ffacdd9b640 1 -- 172.21.15.167:0/4118980630 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffad00aa1f0 con 0x7ffad00a96b0 2024-03-25T18:24:46.031 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.029+0000 7ffacdd9b640 1 -- 172.21.15.167:0/4118980630 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3461290971 0 0) 0x7ffad00a9d60 con 0x7ffad0142a20 2024-03-25T18:24:46.031 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.029+0000 7ffacdd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:24:46.031 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.029+0000 7ffacdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:46.031 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.029+0000 7ffacdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:46.031 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.029+0000 7ffacdd9b640 10 monclient(hunting): my global_id is 4128 2024-03-25T18:24:46.031 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.029+0000 7ffacdd9b640 1 -- 172.21.15.167:0/4118980630 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffad013ff10 con 0x7ffad0142a20 2024-03-25T18:24:46.033 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.032+0000 7ffacdd9b640 1 -- 172.21.15.167:0/4118980630 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 887302763 0 0) 0x7ffad00aa1f0 con 0x7ffad00a96b0 2024-03-25T18:24:46.034 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.032+0000 7ffacdd9b640 1 -- 172.21.15.167:0/4118980630 >> v1:172.21.15.155:6790/0 conn(0x7ffad00b4700 legacy=0x7ffad0142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:46.034 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.032+0000 7ffacdd9b640 1 -- 172.21.15.167:0/4118980630 >> v1:172.21.15.155:6789/0 conn(0x7ffad0142a20 legacy=0x7ffad013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:46.034 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.032+0000 7ffacdd9b640 10 monclient: _finish_hunting 0 2024-03-25T18:24:46.034 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.032+0000 7ffacdd9b640 1 monclient: found mon.noname-c 2024-03-25T18:24:46.034 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.032+0000 7ffacdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:46.034 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.032+0000 7ffacdd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.167:6789/0 2024-03-25T18:24:46.034 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.032+0000 7ffacdd9b640 1 -- 172.21.15.167:0/4118980630 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffad00b4ae0 con 0x7ffad00a96b0 2024-03-25T18:24:46.034 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.032+0000 7ffacdd9b640 10 monclient: _finish_auth 0 2024-03-25T18:24:46.034 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.032+0000 7ffacdd9b640 10 monclient: _check_auth_tickets 2024-03-25T18:24:46.034 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.032+0000 7ffacdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:46.035 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.032+0000 7ffacdd9b640 1 -- 172.21.15.167:0/4118980630 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7ffac80019f0 con 0x7ffad00a96b0 2024-03-25T18:24:46.035 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.032+0000 7ffad4e82640 5 monclient: authenticate success, global_id 4117 2024-03-25T18:24:46.035 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.032+0000 7ffad4e82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:46.035 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.032+0000 7ffacdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:46.035 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.032+0000 7ffacdd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:46.035 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.032+0000 7ffacdd9b640 10 monclient: dump: 2024-03-25T18:24:46.035 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-03-25T18:24:46.035 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:46.035 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:46.035 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:46.035 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:46.035 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 3 2024-03-25T18:24:46.035 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:46.035 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:46.035 INFO:teuthology.orchestra.run.smithi167.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:46.035 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:24:46.036 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.032+0000 7ffad4e82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:46.036 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-03-25T18:24:46.035+0000 7f1820e2b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T18:24:46.060 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.059+0000 7ffacdd9b640 1 -- 172.21.15.167:0/4118980630 <== mon.1 v1:172.21.15.167:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffac8001b50 con 0x7ffad00a96b0 2024-03-25T18:24:46.060 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.059+0000 7ffacdd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:46.060 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.059+0000 7ffacdd9b640 1 -- 172.21.15.167:0/4118980630 <== mon.1 v1:172.21.15.167:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7ffac8001fa0 con 0x7ffad00a96b0 2024-03-25T18:24:46.060 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.059+0000 7ffacdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:46.060 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.059+0000 7ffacdd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:46.060 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.059+0000 7ffacdd9b640 10 monclient: dump: 2024-03-25T18:24:46.060 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-03-25T18:24:46.060 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:46.060 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:46.060 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:46.061 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:46.061 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 3 2024-03-25T18:24:46.061 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:46.061 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:46.061 INFO:teuthology.orchestra.run.smithi167.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:46.061 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:24:46.061 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.059+0000 7ffad4e82640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:46.061 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.059+0000 7ffad4e82640 4 set_mon_vals no callback set 2024-03-25T18:24:46.063 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.060+0000 7ffad4e82640 10 monclient: shutdown 2024-03-25T18:24:46.063 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.060+0000 7ffad4e82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:46.063 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.060+0000 7ffad4e82640 1 -- 172.21.15.167:0/4118980630 >> v1:172.21.15.167:6789/0 conn(0x7ffad00a96b0 legacy=0x7ffad013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:46.064 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.060+0000 7ffad4e82640 1 -- 172.21.15.167:0/4118980630 shutdown_connections 2024-03-25T18:24:46.064 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.060+0000 7ffad4e82640 1 -- 172.21.15.167:0/4118980630 >> 172.21.15.167:0/4118980630 conn(0x7ffad00668e0 msgr2=0x7ffad0066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:46.064 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.060+0000 7ffad4e82640 1 -- 172.21.15.167:0/4118980630 shutdown_connections 2024-03-25T18:24:46.064 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.060+0000 7ffad4e82640 1 -- 172.21.15.167:0/4118980630 wait complete. 2024-03-25T18:24:46.064 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.061+0000 7ffad4e82640 10 monclient: build_initial_monmap 2024-03-25T18:24:46.064 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.061+0000 7ffad4e82640 10 monclient: monmap: 2024-03-25T18:24:46.064 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 0 2024-03-25T18:24:46.064 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:46.064 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 0.000000 2024-03-25T18:24:46.064 INFO:teuthology.orchestra.run.smithi167.stderr:created 0.000000 2024-03-25T18:24:46.064 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:46.064 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 1 2024-03-25T18:24:46.064 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:24:46.064 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:24:46.064 INFO:teuthology.orchestra.run.smithi167.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:24:46.064 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:24:46.065 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.061+0000 7ffad4e82640 1 librados: starting msgr at 2024-03-25T18:24:46.065 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.061+0000 7ffad4e82640 1 librados: starting objecter 2024-03-25T18:24:46.065 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.061+0000 7ffad4e82640 1 Processor -- start 2024-03-25T18:24:46.065 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.061+0000 7ffad4e82640 1 -- start start 2024-03-25T18:24:46.065 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.061+0000 7ffad4e82640 1 librados: setting wanted keys 2024-03-25T18:24:46.065 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.061+0000 7ffad4e82640 1 librados: calling monclient init 2024-03-25T18:24:46.065 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.061+0000 7ffad4e82640 10 monclient: init 2024-03-25T18:24:46.065 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.062+0000 7ffad4e82640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:46.065 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.062+0000 7ffad4e82640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:24:46.065 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.062+0000 7ffad4e82640 10 monclient(hunting): picked mon.noname-c con 0x7ffad00a96b0 addr v1:172.21.15.155:6790/0 2024-03-25T18:24:46.065 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.062+0000 7ffad4e82640 10 monclient(hunting): picked mon.noname-a con 0x7ffad00b4700 addr v1:172.21.15.155:6789/0 2024-03-25T18:24:46.065 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.062+0000 7ffad4e82640 10 monclient(hunting): picked mon.noname-b con 0x7ffad0142a20 addr v1:172.21.15.167:6789/0 2024-03-25T18:24:46.065 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.062+0000 7ffad4e82640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffad01396d0 con 0x7ffad00b4700 2024-03-25T18:24:46.065 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.062+0000 7ffad4e82640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffad01d7670 con 0x7ffad0142a20 2024-03-25T18:24:46.066 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.062+0000 7ffad4e82640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffad0137070 con 0x7ffad00a96b0 2024-03-25T18:24:46.066 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.062+0000 7ffad4e82640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:46.066 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.062+0000 7ffad4e82640 10 monclient(hunting): authenticate will time out at 7516.184144s 2024-03-25T18:24:46.066 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.063+0000 7ffac7fff640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7ffad00b4700 0x7ffad0137ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.167:38490/0 (socket says 172.21.15.167:38490) 2024-03-25T18:24:46.066 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.063+0000 7ffac7fff640 1 -- 172.21.15.167:0/3773158182 learned_addr learned my addr 172.21.15.167:0/3773158182 (peer_addr_for_me v1:172.21.15.167:0/0) 2024-03-25T18:24:46.067 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.063+0000 7ffac6ffd640 10 client.?.objecter ms_handle_connect 0x7ffad00b4700 2024-03-25T18:24:46.067 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.063+0000 7ffac6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:46.067 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.065+0000 7ffac6ffd640 1 -- 172.21.15.167:0/3773158182 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4242150993 0 0) 0x7ffad01396d0 con 0x7ffad00b4700 2024-03-25T18:24:46.067 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.065+0000 7ffac6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:46.067 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.065+0000 7ffac6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:46.067 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.065+0000 7ffac6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:46.067 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.065+0000 7ffac6ffd640 10 monclient(hunting): my global_id is 4131 2024-03-25T18:24:46.067 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.065+0000 7ffac6ffd640 1 -- 172.21.15.167:0/3773158182 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffab0004eb0 con 0x7ffad00b4700 2024-03-25T18:24:46.067 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.065+0000 7ffac6ffd640 10 client.?.objecter ms_handle_connect 0x7ffad0142a20 2024-03-25T18:24:46.067 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.065+0000 7ffac6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:46.067 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.066+0000 7ffac6ffd640 10 client.?.objecter ms_handle_connect 0x7ffad00a96b0 2024-03-25T18:24:46.067 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.066+0000 7ffac6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:46.069 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.068+0000 7ffac6ffd640 1 -- 172.21.15.167:0/3773158182 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2856823679 0 0) 0x7ffad0137070 con 0x7ffad00a96b0 2024-03-25T18:24:46.069 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.068+0000 7ffac6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:46.069 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.068+0000 7ffac6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:46.069 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.068+0000 7ffac6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:46.069 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.068+0000 7ffac6ffd640 10 monclient(hunting): my global_id is 4127 2024-03-25T18:24:46.070 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.068+0000 7ffac6ffd640 1 -- 172.21.15.167:0/3773158182 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffad01396d0 con 0x7ffad00a96b0 2024-03-25T18:24:46.071 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.070+0000 7ffac6ffd640 1 -- 172.21.15.167:0/3773158182 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 368802371 0 0) 0x7ffad01d7670 con 0x7ffad0142a20 2024-03-25T18:24:46.071 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.070+0000 7ffac6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:46.071 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.070+0000 7ffac6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:46.071 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.070+0000 7ffac6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:46.071 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.070+0000 7ffac6ffd640 10 monclient(hunting): my global_id is 4120 2024-03-25T18:24:46.071 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.070+0000 7ffac6ffd640 1 -- 172.21.15.167:0/3773158182 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffad0137070 con 0x7ffad0142a20 2024-03-25T18:24:46.074 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.073+0000 7ffac6ffd640 1 -- 172.21.15.167:0/3773158182 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4120490516 0 0) 0x7ffab0004eb0 con 0x7ffad00b4700 2024-03-25T18:24:46.074 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.073+0000 7ffac6ffd640 1 -- 172.21.15.167:0/3773158182 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffad01d7670 con 0x7ffad00b4700 2024-03-25T18:24:46.074 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.073+0000 7ffac6ffd640 1 -- 172.21.15.167:0/3773158182 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7ffab8001a10 con 0x7ffad00b4700 2024-03-25T18:24:46.074 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.073+0000 7ffac6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:46.075 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.073+0000 7ffac6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:24:46.075 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.073+0000 7ffac6ffd640 10 monclient(hunting): dump: 2024-03-25T18:24:46.075 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-03-25T18:24:46.075 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:46.075 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:46.075 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:46.075 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:46.075 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 3 2024-03-25T18:24:46.075 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:46.075 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:46.075 INFO:teuthology.orchestra.run.smithi167.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:46.075 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:24:46.075 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.073+0000 7ffac6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:46.076 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.075+0000 7ffac6ffd640 1 -- 172.21.15.167:0/3773158182 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1363257148 0 0) 0x7ffad01396d0 con 0x7ffad00a96b0 2024-03-25T18:24:46.076 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.075+0000 7ffac6ffd640 1 -- 172.21.15.167:0/3773158182 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffab0004eb0 con 0x7ffad00a96b0 2024-03-25T18:24:46.077 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.075+0000 7ffac6ffd640 1 -- 172.21.15.167:0/3773158182 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7ffac00012f0 con 0x7ffad00a96b0 2024-03-25T18:24:46.077 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.075+0000 7ffac6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:46.077 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.075+0000 7ffac6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:46.077 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.075+0000 7ffac6ffd640 10 monclient(hunting): dump: 2024-03-25T18:24:46.077 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-03-25T18:24:46.077 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:46.077 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:46.077 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:46.077 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:46.077 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 3 2024-03-25T18:24:46.077 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:46.077 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:46.077 INFO:teuthology.orchestra.run.smithi167.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:46.078 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:24:46.078 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.077+0000 7ffac6ffd640 1 -- 172.21.15.167:0/3773158182 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1991166791 0 0) 0x7ffad0137070 con 0x7ffad0142a20 2024-03-25T18:24:46.078 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.077+0000 7ffac6ffd640 1 -- 172.21.15.167:0/3773158182 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffad01396d0 con 0x7ffad0142a20 2024-03-25T18:24:46.078 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.077+0000 7ffac6ffd640 1 -- 172.21.15.167:0/3773158182 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7ffac8001e60 con 0x7ffad0142a20 2024-03-25T18:24:46.078 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.077+0000 7ffac6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:46.079 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.077+0000 7ffac6ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:46.079 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.077+0000 7ffac6ffd640 10 monclient(hunting): dump: 2024-03-25T18:24:46.079 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-03-25T18:24:46.079 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:46.079 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:46.079 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:46.079 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:46.079 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 3 2024-03-25T18:24:46.079 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:46.079 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:46.079 INFO:teuthology.orchestra.run.smithi167.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:46.079 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:24:46.081 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.080+0000 7ffac6ffd640 1 -- 172.21.15.167:0/3773158182 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3563323060 0 0) 0x7ffad01d7670 con 0x7ffad00b4700 2024-03-25T18:24:46.081 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.080+0000 7ffac6ffd640 1 -- 172.21.15.167:0/3773158182 >> v1:172.21.15.155:6790/0 conn(0x7ffad00a96b0 legacy=0x7ffad01d7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:46.081 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.080+0000 7ffac6ffd640 1 -- 172.21.15.167:0/3773158182 >> v1:172.21.15.167:6789/0 conn(0x7ffad0142a20 legacy=0x7ffad01e1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:46.081 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.080+0000 7ffac6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:24:46.081 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.080+0000 7ffac6ffd640 1 monclient: found mon.a 2024-03-25T18:24:46.081 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.080+0000 7ffac6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:46.081 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.080+0000 7ffac6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:24:46.081 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.080+0000 7ffac6ffd640 1 -- 172.21.15.167:0/3773158182 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffad01d82a0 con 0x7ffad00b4700 2024-03-25T18:24:46.082 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.080+0000 7ffac6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:24:46.082 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.080+0000 7ffac6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:24:46.082 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.080+0000 7ffac6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:46.082 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.080+0000 7ffad4e82640 5 monclient: authenticate success, global_id 4131 2024-03-25T18:24:46.082 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.080+0000 7ffad4e82640 10 monclient: _renew_subs 2024-03-25T18:24:46.082 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.080+0000 7ffad4e82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:24:46.082 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.080+0000 7ffad4e82640 1 -- 172.21.15.167:0/3773158182 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffad01d4fe0 con 0x7ffad00b4700 2024-03-25T18:24:46.082 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.080+0000 7ffad4e82640 10 client.4131.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:46.082 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.080+0000 7ffad4e82640 10 monclient: _renew_subs 2024-03-25T18:24:46.082 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.080+0000 7ffad4e82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:24:46.082 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.080+0000 7ffad4e82640 1 -- 172.21.15.167:0/3773158182 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffad01383d0 con 0x7ffad00b4700 2024-03-25T18:24:46.082 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.080+0000 7ffad4e82640 1 librados: init done 2024-03-25T18:24:46.083 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.081+0000 7ffad4e82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:46.083 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.081+0000 7ffad4e82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:46.083 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.081+0000 7ffad4e82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:24:46.083 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.081+0000 7ffad4e82640 1 -- 172.21.15.167:0/3773158182 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffad00b3df0 con 0x7ffad00b4700 2024-03-25T18:24:46.083 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.081+0000 7ffac6ffd640 1 -- 172.21.15.167:0/3773158182 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffab8001c70 con 0x7ffad00b4700 2024-03-25T18:24:46.083 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.081+0000 7ffac6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:46.083 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.081+0000 7ffacd59a640 4 set_mon_vals no callback set 2024-03-25T18:24:46.085 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.083+0000 7ffac6ffd640 1 -- 172.21.15.167:0/3773158182 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7ffab8003a60 con 0x7ffad00b4700 2024-03-25T18:24:46.085 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.083+0000 7ffac6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:46.085 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.083+0000 7ffac6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:46.085 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.083+0000 7ffac6ffd640 10 monclient: dump: 2024-03-25T18:24:46.085 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-03-25T18:24:46.085 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:46.086 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:46.086 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:46.086 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:46.086 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 3 2024-03-25T18:24:46.086 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:46.086 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:46.086 INFO:teuthology.orchestra.run.smithi167.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:46.086 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:24:46.086 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.084+0000 7ffac6ffd640 1 -- 172.21.15.167:0/3773158182 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7ffab8003fd0 con 0x7ffad00b4700 2024-03-25T18:24:46.086 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.084+0000 7ffac6ffd640 1 -- 172.21.15.167:0/3773158182 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 2881727490 0 0) 0x7ffab80067b0 con 0x7ffad00b4700 2024-03-25T18:24:46.086 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.084+0000 7ffac6ffd640 10 client.4131.objecter ms_dispatch 0x7ffad0000b90 osd_map(4..4 src has 1..4) v4 2024-03-25T18:24:46.086 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.084+0000 7ffac6ffd640 3 client.4131.objecter handle_osd_map got epochs [4,4] > 0 2024-03-25T18:24:46.086 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.084+0000 7ffac6ffd640 3 client.4131.objecter handle_osd_map decoding full epoch 4 2024-03-25T18:24:46.086 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.084+0000 7ffac6ffd640 20 client.4131.objecter dump_active .. 0 homeless 2024-03-25T18:24:46.180 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.179+0000 7ffac6ffd640 1 -- 172.21.15.167:0/3773158182 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7ffab80064a0 con 0x7ffad00b4700 2024-03-25T18:24:46.181 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.180+0000 7ffac6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:46.181 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.180+0000 7ffac6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:46.253 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.252+0000 7ffad4e82640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "03747c34-7e23-428a-8e35-b6132f7e8ae7", "id": 2}] 2024-03-25T18:24:46.253 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.252+0000 7ffad4e82640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "03747c34-7e23-428a-8e35-b6132f7e8ae7", "id": 2}] 2024-03-25T18:24:46.253 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.252+0000 7ffad4e82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:24:46.253 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.252+0000 7ffad4e82640 1 -- 172.21.15.167:0/3773158182 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd new", "uuid": "03747c34-7e23-428a-8e35-b6132f7e8ae7", "id": 2} v 0) v1 -- 0x7ffad004f730 con 0x7ffad00b4700 2024-03-25T18:24:46.278 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.278+0000 7ffac6ffd640 1 -- 172.21.15.167:0/3773158182 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "03747c34-7e23-428a-8e35-b6132f7e8ae7", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 810435149 0 216599125) 0x7ffab8004280 con 0x7ffad00b4700 2024-03-25T18:24:46.279 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.278+0000 7ffac6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "03747c34-7e23-428a-8e35-b6132f7e8ae7", "id": 2}] 2024-03-25T18:24:46.279 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.278+0000 7ffac6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:46.279 INFO:teuthology.orchestra.run.smithi167.stdout:2 2024-03-25T18:24:46.280 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.278+0000 7ffad4e82640 10 librados: watch_flush enter 2024-03-25T18:24:46.280 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.278+0000 7ffad4e82640 10 librados: watch_flush exit 2024-03-25T18:24:46.280 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.278+0000 7ffad4e82640 20 client.4131.objecter shutdown clearing up homeless session... 2024-03-25T18:24:46.280 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.278+0000 7ffad4e82640 10 client.4131.objecter successfully canceled tick 2024-03-25T18:24:46.280 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.279+0000 7ffad4e82640 10 monclient: shutdown 2024-03-25T18:24:46.280 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.279+0000 7ffad4e82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:46.280 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.279+0000 7ffad4e82640 1 -- 172.21.15.167:0/3773158182 >> v1:172.21.15.155:6789/0 conn(0x7ffad00b4700 legacy=0x7ffad0137ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:46.280 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.279+0000 7ffad4e82640 1 -- 172.21.15.167:0/3773158182 shutdown_connections 2024-03-25T18:24:46.280 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.279+0000 7ffad4e82640 1 -- 172.21.15.167:0/3773158182 >> 172.21.15.167:0/3773158182 conn(0x7ffad00668e0 msgr2=0x7ffad00b31e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:46.281 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.279+0000 7ffad4e82640 1 -- 172.21.15.167:0/3773158182 shutdown_connections 2024-03-25T18:24:46.281 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.279+0000 7ffad4e82640 1 -- 172.21.15.167:0/3773158182 wait complete. 2024-03-25T18:24:46.281 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.279+0000 7ffad4e82640 1 librados: shutdown 2024-03-25T18:24:46.291 DEBUG:teuthology.orchestra.run.smithi167:> sudo ceph --cluster ceph osd new ba181ae1-e467-47ee-b719-810d7d0b9e45 3 2024-03-25T18:24:46.388 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-25T18:24:46.387+0000 7f91e582f040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T18:24:46.390 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.389+0000 7f25bf455640 10 monclient: get_monmap_and_config 2024-03-25T18:24:46.390 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.389+0000 7f25bf455640 10 monclient: build_initial_monmap 2024-03-25T18:24:46.390 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.389+0000 7f25bf455640 10 monclient: monmap: 2024-03-25T18:24:46.390 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 0 2024-03-25T18:24:46.390 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:46.390 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-03-25T18:24:46.390514+0000 2024-03-25T18:24:46.391 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-03-25T18:24:46.390514+0000 2024-03-25T18:24:46.391 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:46.391 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 3 2024-03-25T18:24:46.391 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:24:46.391 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:24:46.391 INFO:teuthology.orchestra.run.smithi167.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:24:46.391 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:24:46.392 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.390+0000 7f25bf455640 1 Processor -- start 2024-03-25T18:24:46.392 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.390+0000 7f25bf455640 1 -- start start 2024-03-25T18:24:46.392 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.390+0000 7f25bf455640 10 monclient: init 2024-03-25T18:24:46.392 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.390+0000 7f25bf455640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:46.392 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.390+0000 7f25bf455640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:24:46.393 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.390+0000 7f25bf455640 10 monclient(hunting): picked mon.noname-b con 0x7f25b80b4700 addr v1:172.21.15.155:6790/0 2024-03-25T18:24:46.393 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.390+0000 7f25bf455640 10 monclient(hunting): picked mon.noname-a con 0x7f25b8142a20 addr v1:172.21.15.155:6789/0 2024-03-25T18:24:46.393 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.391+0000 7f25bf455640 10 monclient(hunting): picked mon.noname-c con 0x7f25b80a96b0 addr v1:172.21.15.167:6789/0 2024-03-25T18:24:46.393 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.391+0000 7f25bf455640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25b80a9d60 con 0x7f25b8142a20 2024-03-25T18:24:46.393 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.391+0000 7f25bf455640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25b813ff10 con 0x7f25b80a96b0 2024-03-25T18:24:46.393 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.391+0000 7f25bf455640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25b80aa1f0 con 0x7f25b80b4700 2024-03-25T18:24:46.393 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.391+0000 7f25bf455640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:46.393 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.391+0000 7f25bf455640 10 monclient(hunting): authenticate will time out at 7516.512808s 2024-03-25T18:24:46.393 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.391+0000 7f25bd1f1640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f25b80b4700 0x7f25b8142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.167:51948/0 (socket says 172.21.15.167:51948) 2024-03-25T18:24:46.393 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.391+0000 7f25bd1f1640 1 -- 172.21.15.167:0/2207699385 learned_addr learned my addr 172.21.15.167:0/2207699385 (peer_addr_for_me v1:172.21.15.167:0/0) 2024-03-25T18:24:46.395 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.394+0000 7f25affff640 1 -- 172.21.15.167:0/2207699385 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3294070333 0 0) 0x7f25b80aa1f0 con 0x7f25b80b4700 2024-03-25T18:24:46.395 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.394+0000 7f25affff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:46.395 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.394+0000 7f25affff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:46.395 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.394+0000 7f25affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:46.395 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.394+0000 7f25affff640 10 monclient(hunting): my global_id is 4130 2024-03-25T18:24:46.395 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.394+0000 7f25affff640 1 -- 172.21.15.167:0/2207699385 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25a4003520 con 0x7f25b80b4700 2024-03-25T18:24:46.395 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.394+0000 7f25affff640 1 -- 172.21.15.167:0/2207699385 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3076923071 0 0) 0x7f25b813ff10 con 0x7f25b80a96b0 2024-03-25T18:24:46.395 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.394+0000 7f25affff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:46.395 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.394+0000 7f25affff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:46.395 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.394+0000 7f25affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:46.396 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.394+0000 7f25affff640 10 monclient(hunting): my global_id is 4123 2024-03-25T18:24:46.396 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.394+0000 7f25affff640 1 -- 172.21.15.167:0/2207699385 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25b80aa1f0 con 0x7f25b80a96b0 2024-03-25T18:24:46.396 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.394+0000 7f25affff640 1 -- 172.21.15.167:0/2207699385 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1011244164 0 0) 0x7f25b80a9d60 con 0x7f25b8142a20 2024-03-25T18:24:46.396 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.394+0000 7f25affff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:46.396 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.394+0000 7f25affff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:46.396 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.394+0000 7f25affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:46.396 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.394+0000 7f25affff640 10 monclient(hunting): my global_id is 4134 2024-03-25T18:24:46.396 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.394+0000 7f25affff640 1 -- 172.21.15.167:0/2207699385 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25b813ff10 con 0x7f25b8142a20 2024-03-25T18:24:46.398 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.397+0000 7f25affff640 1 -- 172.21.15.167:0/2207699385 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2852894053 0 0) 0x7f25b80aa1f0 con 0x7f25b80a96b0 2024-03-25T18:24:46.398 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.397+0000 7f25affff640 1 -- 172.21.15.167:0/2207699385 >> v1:172.21.15.155:6790/0 conn(0x7f25b80b4700 legacy=0x7f25b8142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:46.399 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.397+0000 7f25affff640 1 -- 172.21.15.167:0/2207699385 >> v1:172.21.15.155:6789/0 conn(0x7f25b8142a20 legacy=0x7f25b813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:46.399 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.397+0000 7f25affff640 10 monclient: _finish_hunting 0 2024-03-25T18:24:46.399 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.397+0000 7f25affff640 1 monclient: found mon.noname-c 2024-03-25T18:24:46.399 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.397+0000 7f25affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:46.399 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.397+0000 7f25affff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.167:6789/0 2024-03-25T18:24:46.399 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.397+0000 7f25affff640 1 -- 172.21.15.167:0/2207699385 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25b80b4ae0 con 0x7f25b80a96b0 2024-03-25T18:24:46.399 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.397+0000 7f25affff640 10 monclient: _finish_auth 0 2024-03-25T18:24:46.399 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.397+0000 7f25affff640 10 monclient: _check_auth_tickets 2024-03-25T18:24:46.399 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.397+0000 7f25affff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:46.399 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.397+0000 7f25affff640 1 -- 172.21.15.167:0/2207699385 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f25b0001970 con 0x7f25b80a96b0 2024-03-25T18:24:46.399 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.397+0000 7f25affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:46.399 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.397+0000 7f25affff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:46.399 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.397+0000 7f25affff640 10 monclient: dump: 2024-03-25T18:24:46.399 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-03-25T18:24:46.399 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:46.400 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:46.400 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:46.400 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:46.400 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 3 2024-03-25T18:24:46.400 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:46.400 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:46.400 INFO:teuthology.orchestra.run.smithi167.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:46.400 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:24:46.400 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.397+0000 7f25bf455640 5 monclient: authenticate success, global_id 4123 2024-03-25T18:24:46.400 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.397+0000 7f25bf455640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:46.401 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.399+0000 7f25affff640 1 -- 172.21.15.167:0/2207699385 <== mon.1 v1:172.21.15.167:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f25b0001ad0 con 0x7f25b80a96b0 2024-03-25T18:24:46.401 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.399+0000 7f25affff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:46.401 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.399+0000 7f25affff640 1 -- 172.21.15.167:0/2207699385 <== mon.1 v1:172.21.15.167:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f25b0001ee0 con 0x7f25b80a96b0 2024-03-25T18:24:46.401 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.399+0000 7f25affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:46.401 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.399+0000 7f25affff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:46.401 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.399+0000 7f25affff640 10 monclient: dump: 2024-03-25T18:24:46.401 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-03-25T18:24:46.401 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:46.401 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:46.401 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:46.401 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:46.401 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 3 2024-03-25T18:24:46.402 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:46.402 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:46.402 INFO:teuthology.orchestra.run.smithi167.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:46.402 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:24:46.402 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.399+0000 7f25bf455640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:46.402 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.399+0000 7f25bf455640 4 set_mon_vals no callback set 2024-03-25T18:24:46.403 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.400+0000 7f25bf455640 10 monclient: shutdown 2024-03-25T18:24:46.403 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.400+0000 7f25bf455640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:46.403 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.400+0000 7f25bf455640 1 -- 172.21.15.167:0/2207699385 >> v1:172.21.15.167:6789/0 conn(0x7f25b80a96b0 legacy=0x7f25b813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:46.404 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.400+0000 7f25bf455640 1 -- 172.21.15.167:0/2207699385 shutdown_connections 2024-03-25T18:24:46.404 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.400+0000 7f25bf455640 1 -- 172.21.15.167:0/2207699385 >> 172.21.15.167:0/2207699385 conn(0x7f25b80668e0 msgr2=0x7f25b8066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:46.404 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.400+0000 7f25bf455640 1 -- 172.21.15.167:0/2207699385 shutdown_connections 2024-03-25T18:24:46.404 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.401+0000 7f25bf455640 1 -- 172.21.15.167:0/2207699385 wait complete. 2024-03-25T18:24:46.404 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.401+0000 7f25bf455640 10 monclient: build_initial_monmap 2024-03-25T18:24:46.404 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.401+0000 7f25bf455640 10 monclient: monmap: 2024-03-25T18:24:46.404 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 0 2024-03-25T18:24:46.404 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:46.404 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 0.000000 2024-03-25T18:24:46.404 INFO:teuthology.orchestra.run.smithi167.stderr:created 0.000000 2024-03-25T18:24:46.404 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:46.404 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 1 2024-03-25T18:24:46.404 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:24:46.404 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:24:46.404 INFO:teuthology.orchestra.run.smithi167.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:24:46.405 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:24:46.405 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.402+0000 7f25bf455640 1 librados: starting msgr at 2024-03-25T18:24:46.405 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.402+0000 7f25bf455640 1 librados: starting objecter 2024-03-25T18:24:46.405 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.402+0000 7f25bf455640 1 Processor -- start 2024-03-25T18:24:46.405 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.402+0000 7f25bf455640 1 -- start start 2024-03-25T18:24:46.405 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.402+0000 7f25bf455640 1 librados: setting wanted keys 2024-03-25T18:24:46.405 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.402+0000 7f25bf455640 1 librados: calling monclient init 2024-03-25T18:24:46.405 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.402+0000 7f25bf455640 10 monclient: init 2024-03-25T18:24:46.406 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.403+0000 7f25bf455640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:46.406 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.403+0000 7f25bf455640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:24:46.406 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.403+0000 7f25bf455640 10 monclient(hunting): picked mon.noname-a con 0x7f25b80a96b0 addr v1:172.21.15.155:6789/0 2024-03-25T18:24:46.407 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.403+0000 7f25bf455640 10 monclient(hunting): picked mon.noname-c con 0x7f25b80b4700 addr v1:172.21.15.155:6790/0 2024-03-25T18:24:46.407 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.403+0000 7f25bf455640 10 monclient(hunting): picked mon.noname-b con 0x7f25b8142a20 addr v1:172.21.15.167:6789/0 2024-03-25T18:24:46.407 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.403+0000 7f25bf455640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25b813c3c0 con 0x7f25b80a96b0 2024-03-25T18:24:46.407 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.403+0000 7f25bf455640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25b80ac100 con 0x7f25b8142a20 2024-03-25T18:24:46.407 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.403+0000 7f25bf455640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25b81d7280 con 0x7f25b80b4700 2024-03-25T18:24:46.407 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.403+0000 7f25bf455640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:46.407 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.403+0000 7f25bf455640 10 monclient(hunting): authenticate will time out at 7516.525266s 2024-03-25T18:24:46.407 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.403+0000 7f25bc9f0640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f25b80b4700 0x7f25b81d7a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.167:51958/0 (socket says 172.21.15.167:51958) 2024-03-25T18:24:46.407 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.403+0000 7f25bc9f0640 1 -- 172.21.15.167:0/1816525798 learned_addr learned my addr 172.21.15.167:0/1816525798 (peer_addr_for_me v1:172.21.15.167:0/0) 2024-03-25T18:24:46.407 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.404+0000 7f25adffb640 10 client.?.objecter ms_handle_connect 0x7f25b8142a20 2024-03-25T18:24:46.407 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.404+0000 7f25adffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:46.407 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.404+0000 7f25adffb640 10 client.?.objecter ms_handle_connect 0x7f25b80b4700 2024-03-25T18:24:46.407 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.404+0000 7f25adffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:46.407 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.404+0000 7f25adffb640 10 client.?.objecter ms_handle_connect 0x7f25b80a96b0 2024-03-25T18:24:46.407 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.404+0000 7f25adffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:46.409 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.406+0000 7f25adffb640 1 -- 172.21.15.167:0/1816525798 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1853732552 0 0) 0x7f25b81d7280 con 0x7f25b80b4700 2024-03-25T18:24:46.409 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.406+0000 7f25adffb640 10 monclient(hunting): _negotiate 2024-03-25T18:24:46.409 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.406+0000 7f25adffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:46.409 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.406+0000 7f25adffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:46.409 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.406+0000 7f25adffb640 10 monclient(hunting): my global_id is 4133 2024-03-25T18:24:46.409 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.406+0000 7f25adffb640 1 -- 172.21.15.167:0/1816525798 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f259c004eb0 con 0x7f25b80b4700 2024-03-25T18:24:46.409 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.406+0000 7f25adffb640 1 -- 172.21.15.167:0/1816525798 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1554360213 0 0) 0x7f25b813c3c0 con 0x7f25b80a96b0 2024-03-25T18:24:46.409 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.406+0000 7f25adffb640 10 monclient(hunting): _negotiate 2024-03-25T18:24:46.409 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.406+0000 7f25adffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:46.409 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.406+0000 7f25adffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:46.409 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.406+0000 7f25adffb640 10 monclient(hunting): my global_id is 4137 2024-03-25T18:24:46.409 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.406+0000 7f25adffb640 1 -- 172.21.15.167:0/1816525798 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25b81d7280 con 0x7f25b80a96b0 2024-03-25T18:24:46.409 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.406+0000 7f25adffb640 1 -- 172.21.15.167:0/1816525798 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2051512630 0 0) 0x7f25b80ac100 con 0x7f25b8142a20 2024-03-25T18:24:46.409 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.406+0000 7f25adffb640 10 monclient(hunting): _negotiate 2024-03-25T18:24:46.410 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.406+0000 7f25adffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:46.410 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.406+0000 7f25adffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:46.410 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.406+0000 7f25adffb640 10 monclient(hunting): my global_id is 4126 2024-03-25T18:24:46.410 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.406+0000 7f25adffb640 1 -- 172.21.15.167:0/1816525798 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25b813c3c0 con 0x7f25b8142a20 2024-03-25T18:24:46.413 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.411+0000 7f25adffb640 1 -- 172.21.15.167:0/1816525798 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2318946434 0 0) 0x7f25b813c3c0 con 0x7f25b8142a20 2024-03-25T18:24:46.413 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.411+0000 7f25adffb640 1 -- 172.21.15.167:0/1816525798 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25b80ac100 con 0x7f25b8142a20 2024-03-25T18:24:46.413 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.411+0000 7f25adffb640 1 -- 172.21.15.167:0/1816525798 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f25b0001de0 con 0x7f25b8142a20 2024-03-25T18:24:46.413 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.411+0000 7f25adffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:46.413 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.411+0000 7f25adffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:46.413 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.411+0000 7f25adffb640 10 monclient(hunting): dump: 2024-03-25T18:24:46.413 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-03-25T18:24:46.413 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:46.413 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:46.413 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:46.413 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:46.413 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 3 2024-03-25T18:24:46.413 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:46.413 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:46.414 INFO:teuthology.orchestra.run.smithi167.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:46.414 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:24:46.414 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.411+0000 7f25adffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:46.414 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.411+0000 7f25adffb640 1 -- 172.21.15.167:0/1816525798 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1680619038 0 0) 0x7f259c004eb0 con 0x7f25b80b4700 2024-03-25T18:24:46.414 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.411+0000 7f25adffb640 1 -- 172.21.15.167:0/1816525798 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25b813c3c0 con 0x7f25b80b4700 2024-03-25T18:24:46.414 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.411+0000 7f25adffb640 1 -- 172.21.15.167:0/1816525798 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f25a8001990 con 0x7f25b80b4700 2024-03-25T18:24:46.414 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.411+0000 7f25adffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:46.414 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.411+0000 7f25adffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:46.414 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.411+0000 7f25adffb640 10 monclient(hunting): dump: 2024-03-25T18:24:46.414 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-03-25T18:24:46.414 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:46.414 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:46.414 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:46.414 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:46.415 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 3 2024-03-25T18:24:46.415 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:46.415 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:46.415 INFO:teuthology.orchestra.run.smithi167.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:46.415 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:24:46.415 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.411+0000 7f25adffb640 1 -- 172.21.15.167:0/1816525798 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3337529903 0 0) 0x7f25b81d7280 con 0x7f25b80a96b0 2024-03-25T18:24:46.415 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.411+0000 7f25adffb640 1 -- 172.21.15.167:0/1816525798 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f259c004eb0 con 0x7f25b80a96b0 2024-03-25T18:24:46.415 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.411+0000 7f25adffb640 1 -- 172.21.15.167:0/1816525798 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f25b40018d0 con 0x7f25b80a96b0 2024-03-25T18:24:46.415 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.411+0000 7f25adffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:46.415 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.411+0000 7f25adffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:46.415 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.411+0000 7f25adffb640 10 monclient(hunting): dump: 2024-03-25T18:24:46.415 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-03-25T18:24:46.415 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:46.415 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:46.415 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:46.415 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:46.416 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 3 2024-03-25T18:24:46.416 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:46.416 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:46.416 INFO:teuthology.orchestra.run.smithi167.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:46.416 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:24:46.417 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.414+0000 7f25adffb640 1 -- 172.21.15.167:0/1816525798 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2657480880 0 0) 0x7f25b813c3c0 con 0x7f25b80b4700 2024-03-25T18:24:46.417 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.414+0000 7f25adffb640 1 -- 172.21.15.167:0/1816525798 >> v1:172.21.15.167:6789/0 conn(0x7f25b8142a20 legacy=0x7f25b81e1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:46.417 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.414+0000 7f25adffb640 1 -- 172.21.15.167:0/1816525798 >> v1:172.21.15.155:6789/0 conn(0x7f25b80a96b0 legacy=0x7f25b813bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:46.417 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.414+0000 7f25adffb640 10 monclient: _finish_hunting 0 2024-03-25T18:24:46.417 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.414+0000 7f25adffb640 1 monclient: found mon.c 2024-03-25T18:24:46.417 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.414+0000 7f25adffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:46.417 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.414+0000 7f25adffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:24:46.417 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.414+0000 7f25adffb640 1 -- 172.21.15.167:0/1816525798 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25b81d8150 con 0x7f25b80b4700 2024-03-25T18:24:46.417 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.414+0000 7f25adffb640 10 monclient: _finish_auth 0 2024-03-25T18:24:46.417 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.414+0000 7f25adffb640 10 monclient: _check_auth_tickets 2024-03-25T18:24:46.418 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.414+0000 7f25adffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:46.418 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.414+0000 7f25bf455640 5 monclient: authenticate success, global_id 4133 2024-03-25T18:24:46.418 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.414+0000 7f25bf455640 10 monclient: _renew_subs 2024-03-25T18:24:46.418 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.414+0000 7f25bf455640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:24:46.418 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.414+0000 7f25bf455640 1 -- 172.21.15.167:0/1816525798 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25b81d7520 con 0x7f25b80b4700 2024-03-25T18:24:46.418 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.414+0000 7f25bf455640 10 client.4133.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:46.418 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.414+0000 7f25bf455640 10 monclient: _renew_subs 2024-03-25T18:24:46.418 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.414+0000 7f25bf455640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:24:46.418 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.414+0000 7f25bf455640 1 -- 172.21.15.167:0/1816525798 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f25b813b1d0 con 0x7f25b80b4700 2024-03-25T18:24:46.418 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.414+0000 7f25bf455640 1 librados: init done 2024-03-25T18:24:46.418 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.415+0000 7f25bf455640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:46.418 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.415+0000 7f25bf455640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:46.418 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.415+0000 7f25bf455640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:24:46.418 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.415+0000 7f25bf455640 1 -- 172.21.15.167:0/1816525798 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25b804f730 con 0x7f25b80b4700 2024-03-25T18:24:46.439 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-03-25T18:24:46.439+0000 7f1820e2b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T18:24:46.465 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.463+0000 7f25adffb640 1 -- 172.21.15.167:0/1816525798 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f25a8003610 con 0x7f25b80b4700 2024-03-25T18:24:46.465 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.463+0000 7f25adffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:46.465 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.463+0000 7f25adffb640 1 -- 172.21.15.167:0/1816525798 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f25a8003a20 con 0x7f25b80b4700 2024-03-25T18:24:46.465 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.464+0000 7f25adffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:46.465 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.464+0000 7f25adffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:46.465 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.464+0000 7f25adffb640 10 monclient: dump: 2024-03-25T18:24:46.465 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-03-25T18:24:46.465 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:46.465 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:46.465 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:46.465 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:46.465 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 3 2024-03-25T18:24:46.466 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:46.466 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:46.466 INFO:teuthology.orchestra.run.smithi167.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:46.466 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:24:46.466 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.464+0000 7f25af7fe640 4 set_mon_vals no callback set 2024-03-25T18:24:46.466 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.464+0000 7f25adffb640 1 -- 172.21.15.167:0/1816525798 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f25a8003fd0 con 0x7f25b80b4700 2024-03-25T18:24:46.466 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.464+0000 7f25adffb640 1 -- 172.21.15.167:0/1816525798 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 4085024843 0 0) 0x7f25a8006700 con 0x7f25b80b4700 2024-03-25T18:24:46.466 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.464+0000 7f25adffb640 10 client.4133.objecter ms_dispatch 0x7f25b8000b90 osd_map(5..5 src has 1..5) v4 2024-03-25T18:24:46.466 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.464+0000 7f25adffb640 3 client.4133.objecter handle_osd_map got epochs [5,5] > 0 2024-03-25T18:24:46.466 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.464+0000 7f25adffb640 3 client.4133.objecter handle_osd_map decoding full epoch 5 2024-03-25T18:24:46.466 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.464+0000 7f25adffb640 20 client.4133.objecter dump_active .. 0 homeless 2024-03-25T18:24:46.530 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-25T18:24:46.528+0000 7f91e582f040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T18:24:46.577 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-03-25T18:24:46.576+0000 7f1820e2b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T18:24:46.638 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.637+0000 7f25adffb640 1 -- 172.21.15.167:0/1816525798 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f25a8006380 con 0x7f25b80b4700 2024-03-25T18:24:46.638 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.637+0000 7f25adffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:46.638 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.637+0000 7f25adffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:46.689 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-03-25T18:24:46.689+0000 7f1820e2b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T18:24:46.709 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.708+0000 7f25bf455640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ba181ae1-e467-47ee-b719-810d7d0b9e45", "id": 3}] 2024-03-25T18:24:46.710 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.708+0000 7f25bf455640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ba181ae1-e467-47ee-b719-810d7d0b9e45", "id": 3}] 2024-03-25T18:24:46.710 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.708+0000 7f25bf455640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:24:46.710 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.708+0000 7f25bf455640 1 -- 172.21.15.167:0/1816525798 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd new", "uuid": "ba181ae1-e467-47ee-b719-810d7d0b9e45", "id": 3} v 0) v1 -- 0x7f25b8069f30 con 0x7f25b80b4700 2024-03-25T18:24:46.735 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-03-25T18:24:46.734+0000 7f1820e2b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T18:24:46.830 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.829+0000 7f25adffb640 1 -- 172.21.15.167:0/1816525798 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ba181ae1-e467-47ee-b719-810d7d0b9e45", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 2848609755 0 525046306) 0x7f25a8004280 con 0x7f25b80b4700 2024-03-25T18:24:46.830 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.829+0000 7f25adffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ba181ae1-e467-47ee-b719-810d7d0b9e45", "id": 3}] 2024-03-25T18:24:46.830 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.829+0000 7f25adffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:46.831 INFO:teuthology.orchestra.run.smithi167.stdout:3 2024-03-25T18:24:46.832 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.830+0000 7f25bf455640 10 librados: watch_flush enter 2024-03-25T18:24:46.832 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.830+0000 7f25bf455640 10 librados: watch_flush exit 2024-03-25T18:24:46.832 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.830+0000 7f25bf455640 20 client.4133.objecter shutdown clearing up homeless session... 2024-03-25T18:24:46.832 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.830+0000 7f25bf455640 10 client.4133.objecter successfully canceled tick 2024-03-25T18:24:46.832 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.830+0000 7f25bf455640 10 monclient: shutdown 2024-03-25T18:24:46.832 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.830+0000 7f25bf455640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:46.832 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.830+0000 7f25bf455640 1 -- 172.21.15.167:0/1816525798 >> v1:172.21.15.155:6790/0 conn(0x7f25b80b4700 legacy=0x7f25b81d7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:46.832 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.830+0000 7f25bf455640 1 -- 172.21.15.167:0/1816525798 shutdown_connections 2024-03-25T18:24:46.832 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.830+0000 7f25bf455640 1 -- 172.21.15.167:0/1816525798 >> 172.21.15.167:0/1816525798 conn(0x7f25b80668e0 msgr2=0x7f25b813d320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:46.832 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.830+0000 7f25bf455640 1 -- 172.21.15.167:0/1816525798 shutdown_connections 2024-03-25T18:24:46.832 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.830+0000 7f25bf455640 1 -- 172.21.15.167:0/1816525798 wait complete. 2024-03-25T18:24:46.832 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.830+0000 7f25bf455640 1 librados: shutdown 2024-03-25T18:24:46.841 DEBUG:teuthology.orchestra.run.smithi167:> sudo ceph --cluster ceph osd new 5964583a-b9a0-424e-b387-589aa8f8b0f8 4 2024-03-25T18:24:46.850 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-03-25T18:24:46.849+0000 7f1820e2b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:24:46.941 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.940+0000 7efe082d3640 10 monclient: get_monmap_and_config 2024-03-25T18:24:46.941 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.940+0000 7efe082d3640 10 monclient: build_initial_monmap 2024-03-25T18:24:46.941 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.940+0000 7efe082d3640 10 monclient: monmap: 2024-03-25T18:24:46.941 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 0 2024-03-25T18:24:46.941 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:46.941 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-03-25T18:24:46.941032+0000 2024-03-25T18:24:46.941 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-03-25T18:24:46.941032+0000 2024-03-25T18:24:46.942 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:46.942 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 3 2024-03-25T18:24:46.942 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:24:46.942 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:24:46.942 INFO:teuthology.orchestra.run.smithi167.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:24:46.942 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:24:46.942 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.940+0000 7efe082d3640 1 Processor -- start 2024-03-25T18:24:46.942 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.940+0000 7efe082d3640 1 -- start start 2024-03-25T18:24:46.942 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.940+0000 7efe082d3640 10 monclient: init 2024-03-25T18:24:46.943 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-25T18:24:46.941+0000 7f91e582f040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T18:24:46.944 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.941+0000 7efe082d3640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:46.944 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.941+0000 7efe082d3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:24:46.944 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.941+0000 7efe082d3640 10 monclient(hunting): picked mon.noname-a con 0x7efe000b4700 addr v1:172.21.15.155:6789/0 2024-03-25T18:24:46.944 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.941+0000 7efe082d3640 10 monclient(hunting): picked mon.noname-c con 0x7efe00142a20 addr v1:172.21.15.167:6789/0 2024-03-25T18:24:46.944 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.941+0000 7efe082d3640 10 monclient(hunting): picked mon.noname-b con 0x7efe000a96b0 addr v1:172.21.15.155:6790/0 2024-03-25T18:24:46.944 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.941+0000 7efe082d3640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe000a9d60 con 0x7efe000b4700 2024-03-25T18:24:46.944 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.941+0000 7efe082d3640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe0013ff10 con 0x7efe00142a20 2024-03-25T18:24:46.944 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.941+0000 7efe082d3640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe000aa1f0 con 0x7efe000a96b0 2024-03-25T18:24:46.944 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.941+0000 7efe082d3640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:46.944 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.941+0000 7efe082d3640 10 monclient(hunting): authenticate will time out at 7517.063252s 2024-03-25T18:24:46.944 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.942+0000 7efe06870640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7efe000a96b0 0x7efe0013f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.167:51972/0 (socket says 172.21.15.167:51972) 2024-03-25T18:24:46.944 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.942+0000 7efe06870640 1 -- 172.21.15.167:0/369162614 learned_addr learned my addr 172.21.15.167:0/369162614 (peer_addr_for_me v1:172.21.15.167:0/0) 2024-03-25T18:24:46.945 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.944+0000 7efe0506d640 1 -- 172.21.15.167:0/369162614 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3707015838 0 0) 0x7efe0013ff10 con 0x7efe00142a20 2024-03-25T18:24:46.945 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.944+0000 7efe0506d640 10 monclient(hunting): _negotiate 2024-03-25T18:24:46.946 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.944+0000 7efe0506d640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:46.946 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.944+0000 7efe0506d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:46.946 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.944+0000 7efe0506d640 10 monclient(hunting): my global_id is 4129 2024-03-25T18:24:46.946 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.944+0000 7efe0506d640 1 -- 172.21.15.167:0/369162614 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efde8003520 con 0x7efe00142a20 2024-03-25T18:24:46.946 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.944+0000 7efe0506d640 1 -- 172.21.15.167:0/369162614 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 198620158 0 0) 0x7efe000aa1f0 con 0x7efe000a96b0 2024-03-25T18:24:46.946 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.944+0000 7efe0506d640 10 monclient(hunting): _negotiate 2024-03-25T18:24:46.946 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.944+0000 7efe0506d640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:46.946 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.944+0000 7efe0506d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:46.946 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.944+0000 7efe0506d640 10 monclient(hunting): my global_id is 4136 2024-03-25T18:24:46.946 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.944+0000 7efe0506d640 1 -- 172.21.15.167:0/369162614 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe0013ff10 con 0x7efe000a96b0 2024-03-25T18:24:46.946 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.944+0000 7efe0506d640 1 -- 172.21.15.167:0/369162614 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 955348322 0 0) 0x7efe000a9d60 con 0x7efe000b4700 2024-03-25T18:24:46.946 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.944+0000 7efe0506d640 10 monclient(hunting): _negotiate 2024-03-25T18:24:46.946 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.944+0000 7efe0506d640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:46.947 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.944+0000 7efe0506d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:46.947 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.944+0000 7efe0506d640 10 monclient(hunting): my global_id is 4140 2024-03-25T18:24:46.947 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.944+0000 7efe0506d640 1 -- 172.21.15.167:0/369162614 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe000aa1f0 con 0x7efe000b4700 2024-03-25T18:24:46.948 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.947+0000 7efe0506d640 1 -- 172.21.15.167:0/369162614 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 78936045 0 0) 0x7efde8003520 con 0x7efe00142a20 2024-03-25T18:24:46.948 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.947+0000 7efe0506d640 1 -- 172.21.15.167:0/369162614 >> v1:172.21.15.155:6790/0 conn(0x7efe000a96b0 legacy=0x7efe0013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:46.948 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.947+0000 7efe0506d640 1 -- 172.21.15.167:0/369162614 >> v1:172.21.15.155:6789/0 conn(0x7efe000b4700 legacy=0x7efe00142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:46.949 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.947+0000 7efe0506d640 10 monclient: _finish_hunting 0 2024-03-25T18:24:46.949 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.947+0000 7efe0506d640 1 monclient: found mon.noname-c 2024-03-25T18:24:46.949 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.947+0000 7efe0506d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:46.949 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.947+0000 7efe0506d640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.167:6789/0 2024-03-25T18:24:46.949 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.947+0000 7efe0506d640 1 -- 172.21.15.167:0/369162614 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe000b4ae0 con 0x7efe00142a20 2024-03-25T18:24:46.949 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.947+0000 7efe0506d640 10 monclient: _finish_auth 0 2024-03-25T18:24:46.949 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.947+0000 7efe0506d640 10 monclient: _check_auth_tickets 2024-03-25T18:24:46.949 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.947+0000 7efe0506d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:46.949 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.947+0000 7efe0506d640 1 -- 172.21.15.167:0/369162614 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7efdf80015c0 con 0x7efe00142a20 2024-03-25T18:24:46.949 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.947+0000 7efe082d3640 5 monclient: authenticate success, global_id 4129 2024-03-25T18:24:46.949 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.947+0000 7efe082d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:46.949 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.947+0000 7efe0506d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:46.949 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.947+0000 7efe0506d640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:46.949 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.947+0000 7efe0506d640 10 monclient: dump: 2024-03-25T18:24:46.949 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-03-25T18:24:46.949 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:46.950 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:46.950 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:46.950 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:46.950 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 3 2024-03-25T18:24:46.950 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:46.950 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:46.950 INFO:teuthology.orchestra.run.smithi167.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:46.950 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:24:46.950 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.947+0000 7efe082d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:46.951 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.949+0000 7efe0506d640 1 -- 172.21.15.167:0/369162614 <== mon.1 v1:172.21.15.167:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efdf8001720 con 0x7efe00142a20 2024-03-25T18:24:46.951 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.949+0000 7efe0506d640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:46.951 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.949+0000 7efe0506d640 1 -- 172.21.15.167:0/369162614 <== mon.1 v1:172.21.15.167:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7efdf8001b30 con 0x7efe00142a20 2024-03-25T18:24:46.951 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.949+0000 7efe0506d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:46.951 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.949+0000 7efe0506d640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:46.951 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.949+0000 7efe0506d640 10 monclient: dump: 2024-03-25T18:24:46.951 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-03-25T18:24:46.952 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:46.952 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:46.952 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:46.952 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:46.952 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 3 2024-03-25T18:24:46.952 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:46.952 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:46.952 INFO:teuthology.orchestra.run.smithi167.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:46.952 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:24:46.952 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.949+0000 7efe082d3640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:46.952 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.949+0000 7efe082d3640 4 set_mon_vals no callback set 2024-03-25T18:24:46.952 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.949+0000 7efe082d3640 10 monclient: shutdown 2024-03-25T18:24:46.952 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.949+0000 7efe082d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:46.952 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.949+0000 7efe082d3640 1 -- 172.21.15.167:0/369162614 >> v1:172.21.15.167:6789/0 conn(0x7efe00142a20 legacy=0x7efe0013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:46.952 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.950+0000 7efe082d3640 1 -- 172.21.15.167:0/369162614 shutdown_connections 2024-03-25T18:24:46.952 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.950+0000 7efe082d3640 1 -- 172.21.15.167:0/369162614 >> 172.21.15.167:0/369162614 conn(0x7efe000668e0 msgr2=0x7efe00066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:46.953 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.950+0000 7efe082d3640 1 -- 172.21.15.167:0/369162614 shutdown_connections 2024-03-25T18:24:46.953 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.950+0000 7efe082d3640 1 -- 172.21.15.167:0/369162614 wait complete. 2024-03-25T18:24:46.953 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.950+0000 7efe082d3640 10 monclient: build_initial_monmap 2024-03-25T18:24:46.953 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.950+0000 7efe082d3640 10 monclient: monmap: 2024-03-25T18:24:46.953 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 0 2024-03-25T18:24:46.953 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:46.953 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 0.000000 2024-03-25T18:24:46.953 INFO:teuthology.orchestra.run.smithi167.stderr:created 0.000000 2024-03-25T18:24:46.953 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:46.953 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 1 2024-03-25T18:24:46.953 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:24:46.953 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:24:46.953 INFO:teuthology.orchestra.run.smithi167.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:24:46.953 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:24:46.954 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.951+0000 7efe082d3640 1 librados: starting msgr at 2024-03-25T18:24:46.954 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.951+0000 7efe082d3640 1 librados: starting objecter 2024-03-25T18:24:46.954 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-03-25T18:24:46.951+0000 7f1820e2b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T18:24:46.956 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.951+0000 7efe082d3640 1 Processor -- start 2024-03-25T18:24:46.956 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.951+0000 7efe082d3640 1 -- start start 2024-03-25T18:24:46.957 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.951+0000 7efe082d3640 1 librados: setting wanted keys 2024-03-25T18:24:46.957 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.951+0000 7efe082d3640 1 librados: calling monclient init 2024-03-25T18:24:46.957 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.951+0000 7efe082d3640 10 monclient: init 2024-03-25T18:24:46.957 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.951+0000 7efe082d3640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:46.957 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.951+0000 7efe082d3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:24:46.957 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.951+0000 7efe082d3640 10 monclient(hunting): picked mon.noname-a con 0x7efe000a96b0 addr v1:172.21.15.155:6789/0 2024-03-25T18:24:46.957 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.951+0000 7efe082d3640 10 monclient(hunting): picked mon.noname-c con 0x7efe000b4700 addr v1:172.21.15.155:6790/0 2024-03-25T18:24:46.957 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.951+0000 7efe082d3640 10 monclient(hunting): picked mon.noname-b con 0x7efe00142a20 addr v1:172.21.15.167:6789/0 2024-03-25T18:24:46.957 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.951+0000 7efe082d3640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe0014eca0 con 0x7efe000a96b0 2024-03-25T18:24:46.957 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.951+0000 7efe082d3640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe001d74d0 con 0x7efe00142a20 2024-03-25T18:24:46.957 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.951+0000 7efe082d3640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe001d8350 con 0x7efe000b4700 2024-03-25T18:24:46.957 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.951+0000 7efe082d3640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:46.957 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.951+0000 7efe082d3640 10 monclient(hunting): authenticate will time out at 7517.073650s 2024-03-25T18:24:46.957 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.952+0000 7efe0586e640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7efe000b4700 0x7efe000acb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.167:51980/0 (socket says 172.21.15.167:51980) 2024-03-25T18:24:46.957 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.952+0000 7efe0586e640 1 -- 172.21.15.167:0/1250205160 learned_addr learned my addr 172.21.15.167:0/1250205160 (peer_addr_for_me v1:172.21.15.167:0/0) 2024-03-25T18:24:46.958 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.952+0000 7efdfeffd640 10 client.?.objecter ms_handle_connect 0x7efe00142a20 2024-03-25T18:24:46.958 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.952+0000 7efdfeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:46.958 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.952+0000 7efdfeffd640 10 client.?.objecter ms_handle_connect 0x7efe000b4700 2024-03-25T18:24:46.958 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.952+0000 7efdfeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:46.958 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.952+0000 7efdfeffd640 10 client.?.objecter ms_handle_connect 0x7efe000a96b0 2024-03-25T18:24:46.958 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.952+0000 7efdfeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:46.958 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.954+0000 7efdfeffd640 1 -- 172.21.15.167:0/1250205160 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4069871658 0 0) 0x7efe001d74d0 con 0x7efe00142a20 2024-03-25T18:24:46.958 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.954+0000 7efdfeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:46.958 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.954+0000 7efdfeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:46.958 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.954+0000 7efdfeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:46.958 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.954+0000 7efdfeffd640 10 monclient(hunting): my global_id is 4132 2024-03-25T18:24:46.958 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.954+0000 7efdfeffd640 1 -- 172.21.15.167:0/1250205160 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdec004eb0 con 0x7efe00142a20 2024-03-25T18:24:46.959 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.954+0000 7efdfeffd640 1 -- 172.21.15.167:0/1250205160 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1794705602 0 0) 0x7efe001d8350 con 0x7efe000b4700 2024-03-25T18:24:46.959 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.954+0000 7efdfeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:46.959 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.954+0000 7efdfeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:46.959 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.954+0000 7efdfeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:46.959 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.954+0000 7efdfeffd640 10 monclient(hunting): my global_id is 4139 2024-03-25T18:24:46.959 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.954+0000 7efdfeffd640 1 -- 172.21.15.167:0/1250205160 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe001d74d0 con 0x7efe000b4700 2024-03-25T18:24:46.959 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.954+0000 7efdfeffd640 1 -- 172.21.15.167:0/1250205160 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 149938924 0 0) 0x7efe000aa1f0 con 0x7efe000a96b0 2024-03-25T18:24:46.959 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.954+0000 7efdfeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:46.959 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.954+0000 7efdfeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:46.959 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.954+0000 7efdfeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:46.960 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.954+0000 7efdfeffd640 10 monclient(hunting): my global_id is 4143 2024-03-25T18:24:46.960 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.954+0000 7efdfeffd640 1 -- 172.21.15.167:0/1250205160 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe001d8350 con 0x7efe000a96b0 2024-03-25T18:24:46.961 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.959+0000 7efdfeffd640 1 -- 172.21.15.167:0/1250205160 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2727034881 0 0) 0x7efdec004eb0 con 0x7efe00142a20 2024-03-25T18:24:46.961 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.959+0000 7efdfeffd640 1 -- 172.21.15.167:0/1250205160 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe000aa1f0 con 0x7efe00142a20 2024-03-25T18:24:46.961 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.959+0000 7efdfeffd640 1 -- 172.21.15.167:0/1250205160 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7efdf4001bd0 con 0x7efe00142a20 2024-03-25T18:24:46.961 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.959+0000 7efdfeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:46.961 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.959+0000 7efdfeffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:46.961 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.959+0000 7efdfeffd640 10 monclient(hunting): dump: 2024-03-25T18:24:46.962 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-03-25T18:24:46.962 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:46.962 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:46.962 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:46.962 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:46.962 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 3 2024-03-25T18:24:46.962 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:46.962 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:46.962 INFO:teuthology.orchestra.run.smithi167.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:46.962 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:24:46.962 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.959+0000 7efdfeffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:46.962 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.959+0000 7efdfeffd640 1 -- 172.21.15.167:0/1250205160 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3812376986 0 0) 0x7efe001d74d0 con 0x7efe000b4700 2024-03-25T18:24:46.962 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.959+0000 7efdfeffd640 1 -- 172.21.15.167:0/1250205160 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efdec004eb0 con 0x7efe000b4700 2024-03-25T18:24:46.962 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.959+0000 7efdfeffd640 1 -- 172.21.15.167:0/1250205160 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7efdf8001a30 con 0x7efe000b4700 2024-03-25T18:24:46.962 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.959+0000 7efdfeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:46.963 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.959+0000 7efdfeffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:46.963 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.959+0000 7efdfeffd640 10 monclient(hunting): dump: 2024-03-25T18:24:46.963 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-03-25T18:24:46.963 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:46.963 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:46.963 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:46.963 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:46.963 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 3 2024-03-25T18:24:46.963 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:46.963 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:46.963 INFO:teuthology.orchestra.run.smithi167.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:46.963 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:24:46.963 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.959+0000 7efdfeffd640 1 -- 172.21.15.167:0/1250205160 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3356956111 0 0) 0x7efe001d8350 con 0x7efe000a96b0 2024-03-25T18:24:46.963 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.959+0000 7efdfeffd640 1 -- 172.21.15.167:0/1250205160 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe001d74d0 con 0x7efe000a96b0 2024-03-25T18:24:46.964 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.959+0000 7efdfeffd640 1 -- 172.21.15.167:0/1250205160 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7efdf0001690 con 0x7efe000a96b0 2024-03-25T18:24:46.964 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.959+0000 7efdfeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:46.964 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.959+0000 7efdfeffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:46.964 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.959+0000 7efdfeffd640 10 monclient(hunting): dump: 2024-03-25T18:24:46.964 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-03-25T18:24:46.964 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:46.964 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:46.964 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:46.964 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:46.964 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 3 2024-03-25T18:24:46.964 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:46.964 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:46.964 INFO:teuthology.orchestra.run.smithi167.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:46.964 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:24:46.966 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.962+0000 7efdfeffd640 1 -- 172.21.15.167:0/1250205160 <== mon.1 v1:172.21.15.167:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 561569848 0 0) 0x7efe000aa1f0 con 0x7efe00142a20 2024-03-25T18:24:46.966 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.962+0000 7efdfeffd640 1 -- 172.21.15.167:0/1250205160 >> v1:172.21.15.155:6790/0 conn(0x7efe000b4700 legacy=0x7efe000acb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:46.966 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.962+0000 7efdfeffd640 1 -- 172.21.15.167:0/1250205160 >> v1:172.21.15.155:6789/0 conn(0x7efe000a96b0 legacy=0x7efe001d7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:46.966 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.962+0000 7efdfeffd640 10 monclient: _finish_hunting 0 2024-03-25T18:24:46.966 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.962+0000 7efdfeffd640 1 monclient: found mon.b 2024-03-25T18:24:46.966 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.962+0000 7efdfeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:46.967 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.962+0000 7efdfeffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:24:46.967 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.962+0000 7efdfeffd640 1 -- 172.21.15.167:0/1250205160 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe000abd90 con 0x7efe00142a20 2024-03-25T18:24:46.967 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.962+0000 7efdfeffd640 10 monclient: _finish_auth 0 2024-03-25T18:24:46.967 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.962+0000 7efdfeffd640 10 monclient: _check_auth_tickets 2024-03-25T18:24:46.967 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.962+0000 7efdfeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:46.967 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.962+0000 7efe082d3640 5 monclient: authenticate success, global_id 4132 2024-03-25T18:24:46.967 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.962+0000 7efe082d3640 10 monclient: _renew_subs 2024-03-25T18:24:46.967 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.962+0000 7efe082d3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:24:46.967 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.962+0000 7efe082d3640 1 -- 172.21.15.167:0/1250205160 --> v1:172.21.15.167:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe001d7720 con 0x7efe00142a20 2024-03-25T18:24:46.967 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.962+0000 7efe082d3640 10 client.4132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:46.967 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.962+0000 7efe082d3640 10 monclient: _renew_subs 2024-03-25T18:24:46.967 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.962+0000 7efe082d3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:24:46.967 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.962+0000 7efe082d3640 1 -- 172.21.15.167:0/1250205160 --> v1:172.21.15.167:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efe000ad240 con 0x7efe00142a20 2024-03-25T18:24:46.968 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.962+0000 7efe082d3640 1 librados: init done 2024-03-25T18:24:46.968 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.963+0000 7efe082d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:46.968 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.963+0000 7efe082d3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:46.968 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.963+0000 7efe082d3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:24:46.968 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:46.963+0000 7efe082d3640 1 -- 172.21.15.167:0/1250205160 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe0004f730 con 0x7efe00142a20 2024-03-25T18:24:46.992 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-25T18:24:46.990+0000 7f91e582f040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T18:24:47.105 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-25T18:24:47.103+0000 7f91e582f040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:24:47.114 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-03-25T18:24:47.113+0000 7f1820e2b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:24:47.158 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-03-25T18:24:47.157+0000 7f1820e2b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T18:24:47.164 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.163+0000 7efdfeffd640 1 -- 172.21.15.167:0/1250205160 <== mon.1 v1:172.21.15.167:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efdf40020e0 con 0x7efe00142a20 2024-03-25T18:24:47.165 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.163+0000 7efdfeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:47.165 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.163+0000 7efdfeffd640 1 -- 172.21.15.167:0/1250205160 <== mon.1 v1:172.21.15.167:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7efdf4004d00 con 0x7efe00142a20 2024-03-25T18:24:47.165 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.163+0000 7efdfeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:47.165 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.163+0000 7efdfeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:47.165 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.163+0000 7efdfeffd640 10 monclient: dump: 2024-03-25T18:24:47.165 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-03-25T18:24:47.165 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:47.165 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:47.165 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:47.165 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:47.165 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 3 2024-03-25T18:24:47.165 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:47.165 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:47.166 INFO:teuthology.orchestra.run.smithi167.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:47.166 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:24:47.166 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.163+0000 7efdfeffd640 1 -- 172.21.15.167:0/1250205160 <== mon.1 v1:172.21.15.167:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7efdf4005230 con 0x7efe00142a20 2024-03-25T18:24:47.166 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.163+0000 7efe0486c640 4 set_mon_vals no callback set 2024-03-25T18:24:47.166 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.163+0000 7efdfeffd640 1 -- 172.21.15.167:0/1250205160 <== mon.1 v1:172.21.15.167:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 2089020396 0 0) 0x7efdf4006a20 con 0x7efe00142a20 2024-03-25T18:24:47.166 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.163+0000 7efdfeffd640 10 client.4132.objecter ms_dispatch 0x7efe00000b90 osd_map(6..6 src has 1..6) v4 2024-03-25T18:24:47.166 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.163+0000 7efdfeffd640 3 client.4132.objecter handle_osd_map got epochs [6,6] > 0 2024-03-25T18:24:47.166 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.163+0000 7efdfeffd640 3 client.4132.objecter handle_osd_map decoding full epoch 6 2024-03-25T18:24:47.166 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.163+0000 7efdfeffd640 20 client.4132.objecter dump_active .. 0 homeless 2024-03-25T18:24:47.166 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.163+0000 7efdfeffd640 1 -- 172.21.15.167:0/1250205160 <== mon.1 v1:172.21.15.167:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7efdf4018dc0 con 0x7efe00142a20 2024-03-25T18:24:47.166 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.163+0000 7efdfeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:47.166 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.163+0000 7efdfeffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:47.204 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-25T18:24:47.202+0000 7f91e582f040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T18:24:47.240 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.239+0000 7efe082d3640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "5964583a-b9a0-424e-b387-589aa8f8b0f8", "id": 4}] 2024-03-25T18:24:47.240 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.239+0000 7efe082d3640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "5964583a-b9a0-424e-b387-589aa8f8b0f8", "id": 4}] 2024-03-25T18:24:47.240 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.239+0000 7efe082d3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:24:47.240 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.239+0000 7efe082d3640 1 -- 172.21.15.167:0/1250205160 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "osd new", "uuid": "5964583a-b9a0-424e-b387-589aa8f8b0f8", "id": 4} v 0) v1 -- 0x7efe000b7470 con 0x7efe00142a20 2024-03-25T18:24:47.318 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-03-25T18:24:47.317+0000 7f1820e2b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T18:24:47.337 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.336+0000 7efdfeffd640 1 -- 172.21.15.167:0/1250205160 <== mon.1 v1:172.21.15.167:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5964583a-b9a0-424e-b387-589aa8f8b0f8", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 118892109 0 1697012583) 0x7efdf4006510 con 0x7efe00142a20 2024-03-25T18:24:47.337 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.336+0000 7efdfeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "5964583a-b9a0-424e-b387-589aa8f8b0f8", "id": 4}] 2024-03-25T18:24:47.337 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.336+0000 7efdfeffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:47.337 INFO:teuthology.orchestra.run.smithi167.stdout:4 2024-03-25T18:24:47.338 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.337+0000 7efe082d3640 10 librados: watch_flush enter 2024-03-25T18:24:47.338 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.337+0000 7efe082d3640 10 librados: watch_flush exit 2024-03-25T18:24:47.338 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.337+0000 7efe082d3640 20 client.4132.objecter shutdown clearing up homeless session... 2024-03-25T18:24:47.339 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.337+0000 7efe082d3640 10 client.4132.objecter successfully canceled tick 2024-03-25T18:24:47.339 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.337+0000 7efe082d3640 10 monclient: shutdown 2024-03-25T18:24:47.339 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.337+0000 7efe082d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:47.339 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.337+0000 7efe082d3640 1 -- 172.21.15.167:0/1250205160 >> v1:172.21.15.167:6789/0 conn(0x7efe00142a20 legacy=0x7efe001d4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:47.339 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.337+0000 7efe082d3640 1 -- 172.21.15.167:0/1250205160 shutdown_connections 2024-03-25T18:24:47.339 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.337+0000 7efe082d3640 1 -- 172.21.15.167:0/1250205160 >> 172.21.15.167:0/1250205160 conn(0x7efe000668e0 msgr2=0x7efe000b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:47.339 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.337+0000 7efe082d3640 1 -- 172.21.15.167:0/1250205160 shutdown_connections 2024-03-25T18:24:47.339 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.337+0000 7efe082d3640 1 -- 172.21.15.167:0/1250205160 wait complete. 2024-03-25T18:24:47.339 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.337+0000 7efe082d3640 1 librados: shutdown 2024-03-25T18:24:47.348 DEBUG:teuthology.orchestra.run.smithi167:> sudo ceph --cluster ceph osd new bfddfbda-7f87-471d-a105-bc2766a64f83 5 2024-03-25T18:24:47.368 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-25T18:24:47.366+0000 7f91e582f040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:24:47.410 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-25T18:24:47.408+0000 7f91e582f040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T18:24:47.490 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.489+0000 7fd979ab3640 10 monclient: get_monmap_and_config 2024-03-25T18:24:47.490 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.489+0000 7fd979ab3640 10 monclient: build_initial_monmap 2024-03-25T18:24:47.490 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.489+0000 7fd979ab3640 10 monclient: monmap: 2024-03-25T18:24:47.490 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 0 2024-03-25T18:24:47.490 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:47.490 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-03-25T18:24:47.490213+0000 2024-03-25T18:24:47.490 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-03-25T18:24:47.490213+0000 2024-03-25T18:24:47.490 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:47.490 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 3 2024-03-25T18:24:47.490 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:24:47.490 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:24:47.491 INFO:teuthology.orchestra.run.smithi167.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:24:47.491 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:24:47.492 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.490+0000 7fd979ab3640 1 Processor -- start 2024-03-25T18:24:47.492 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.490+0000 7fd979ab3640 1 -- start start 2024-03-25T18:24:47.492 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.490+0000 7fd979ab3640 10 monclient: init 2024-03-25T18:24:47.492 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.490+0000 7fd979ab3640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:47.492 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.490+0000 7fd979ab3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:24:47.492 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.490+0000 7fd979ab3640 10 monclient(hunting): picked mon.noname-b con 0x7fd9740b4700 addr v1:172.21.15.155:6790/0 2024-03-25T18:24:47.492 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.490+0000 7fd979ab3640 10 monclient(hunting): picked mon.noname-a con 0x7fd974142a20 addr v1:172.21.15.155:6789/0 2024-03-25T18:24:47.492 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.490+0000 7fd979ab3640 10 monclient(hunting): picked mon.noname-c con 0x7fd9740a96b0 addr v1:172.21.15.167:6789/0 2024-03-25T18:24:47.492 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.490+0000 7fd979ab3640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9740a9d60 con 0x7fd974142a20 2024-03-25T18:24:47.492 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.490+0000 7fd979ab3640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd97413ff10 con 0x7fd9740a96b0 2024-03-25T18:24:47.492 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.490+0000 7fd979ab3640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9740aa1f0 con 0x7fd9740b4700 2024-03-25T18:24:47.492 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.490+0000 7fd979ab3640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:47.493 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.490+0000 7fd979ab3640 10 monclient(hunting): authenticate will time out at 7517.612396s 2024-03-25T18:24:47.493 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.491+0000 7fd9737fe640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fd9740b4700 0x7fd974142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.167:51990/0 (socket says 172.21.15.167:51990) 2024-03-25T18:24:47.493 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.491+0000 7fd9737fe640 1 -- 172.21.15.167:0/3523746668 learned_addr learned my addr 172.21.15.167:0/3523746668 (peer_addr_for_me v1:172.21.15.167:0/0) 2024-03-25T18:24:47.494 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.493+0000 7fd9727fc640 1 -- 172.21.15.167:0/3523746668 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3818175109 0 0) 0x7fd9740a9d60 con 0x7fd974142a20 2024-03-25T18:24:47.495 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.493+0000 7fd9727fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:47.495 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.493+0000 7fd9727fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:47.495 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.493+0000 7fd9727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:47.495 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.493+0000 7fd9727fc640 10 monclient(hunting): my global_id is 4146 2024-03-25T18:24:47.495 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.493+0000 7fd9727fc640 1 -- 172.21.15.167:0/3523746668 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd95c003520 con 0x7fd974142a20 2024-03-25T18:24:47.495 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.493+0000 7fd9727fc640 1 -- 172.21.15.167:0/3523746668 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 204929330 0 0) 0x7fd97413ff10 con 0x7fd9740a96b0 2024-03-25T18:24:47.495 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.493+0000 7fd9727fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:47.495 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.493+0000 7fd9727fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:47.495 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.493+0000 7fd9727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:47.495 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.493+0000 7fd9727fc640 10 monclient(hunting): my global_id is 4135 2024-03-25T18:24:47.495 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.493+0000 7fd9727fc640 1 -- 172.21.15.167:0/3523746668 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9740a9d60 con 0x7fd9740a96b0 2024-03-25T18:24:47.495 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.494+0000 7fd9727fc640 1 -- 172.21.15.167:0/3523746668 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1290647241 0 0) 0x7fd9740aa1f0 con 0x7fd9740b4700 2024-03-25T18:24:47.495 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.494+0000 7fd9727fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:47.495 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.494+0000 7fd9727fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:47.496 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.494+0000 7fd9727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:47.496 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.494+0000 7fd9727fc640 10 monclient(hunting): my global_id is 4142 2024-03-25T18:24:47.496 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.494+0000 7fd9727fc640 1 -- 172.21.15.167:0/3523746668 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd97413ff10 con 0x7fd9740b4700 2024-03-25T18:24:47.498 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.497+0000 7fd9727fc640 1 -- 172.21.15.167:0/3523746668 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1286506333 0 0) 0x7fd9740a9d60 con 0x7fd9740a96b0 2024-03-25T18:24:47.498 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.497+0000 7fd9727fc640 1 -- 172.21.15.167:0/3523746668 >> v1:172.21.15.155:6790/0 conn(0x7fd9740b4700 legacy=0x7fd974142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:47.498 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.497+0000 7fd9727fc640 1 -- 172.21.15.167:0/3523746668 >> v1:172.21.15.155:6789/0 conn(0x7fd974142a20 legacy=0x7fd97413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:47.498 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.497+0000 7fd9727fc640 10 monclient: _finish_hunting 0 2024-03-25T18:24:47.498 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.497+0000 7fd9727fc640 1 monclient: found mon.noname-c 2024-03-25T18:24:47.498 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.497+0000 7fd9727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:47.498 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.497+0000 7fd9727fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.167:6789/0 2024-03-25T18:24:47.499 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.497+0000 7fd9727fc640 1 -- 172.21.15.167:0/3523746668 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9740b4ae0 con 0x7fd9740a96b0 2024-03-25T18:24:47.499 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.497+0000 7fd9727fc640 10 monclient: _finish_auth 0 2024-03-25T18:24:47.499 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.497+0000 7fd9727fc640 10 monclient: _check_auth_tickets 2024-03-25T18:24:47.499 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.497+0000 7fd9727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:47.499 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.497+0000 7fd9727fc640 1 -- 172.21.15.167:0/3523746668 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fd96c0019f0 con 0x7fd9740a96b0 2024-03-25T18:24:47.499 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.497+0000 7fd979ab3640 5 monclient: authenticate success, global_id 4135 2024-03-25T18:24:47.499 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.497+0000 7fd979ab3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:47.499 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.497+0000 7fd9727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:47.499 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.497+0000 7fd9727fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:47.499 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.497+0000 7fd9727fc640 10 monclient: dump: 2024-03-25T18:24:47.499 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-03-25T18:24:47.499 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:47.499 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:47.499 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:47.499 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:47.499 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 3 2024-03-25T18:24:47.500 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:47.500 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:47.500 INFO:teuthology.orchestra.run.smithi167.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:47.500 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:24:47.500 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.497+0000 7fd979ab3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:47.502 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.498+0000 7fd9727fc640 1 -- 172.21.15.167:0/3523746668 <== mon.1 v1:172.21.15.167:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd96c001b50 con 0x7fd9740a96b0 2024-03-25T18:24:47.503 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.498+0000 7fd9727fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:47.503 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.498+0000 7fd9727fc640 1 -- 172.21.15.167:0/3523746668 <== mon.1 v1:172.21.15.167:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fd96c001f60 con 0x7fd9740a96b0 2024-03-25T18:24:47.503 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.498+0000 7fd979ab3640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:47.503 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.498+0000 7fd979ab3640 4 set_mon_vals no callback set 2024-03-25T18:24:47.503 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.498+0000 7fd9727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:47.503 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.498+0000 7fd9727fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:47.503 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.498+0000 7fd9727fc640 10 monclient: dump: 2024-03-25T18:24:47.503 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-03-25T18:24:47.503 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:47.503 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:47.503 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:47.503 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:47.503 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 3 2024-03-25T18:24:47.503 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:47.503 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:47.504 INFO:teuthology.orchestra.run.smithi167.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:47.504 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:24:47.504 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.499+0000 7fd979ab3640 10 monclient: shutdown 2024-03-25T18:24:47.504 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.499+0000 7fd979ab3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:47.504 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.499+0000 7fd979ab3640 1 -- 172.21.15.167:0/3523746668 >> v1:172.21.15.167:6789/0 conn(0x7fd9740a96b0 legacy=0x7fd97413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:47.504 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.499+0000 7fd979ab3640 1 -- 172.21.15.167:0/3523746668 shutdown_connections 2024-03-25T18:24:47.504 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.499+0000 7fd979ab3640 1 -- 172.21.15.167:0/3523746668 >> 172.21.15.167:0/3523746668 conn(0x7fd9740668e0 msgr2=0x7fd974066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:47.504 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.499+0000 7fd979ab3640 1 -- 172.21.15.167:0/3523746668 shutdown_connections 2024-03-25T18:24:47.504 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.499+0000 7fd979ab3640 1 -- 172.21.15.167:0/3523746668 wait complete. 2024-03-25T18:24:47.504 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.500+0000 7fd979ab3640 10 monclient: build_initial_monmap 2024-03-25T18:24:47.504 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.500+0000 7fd979ab3640 10 monclient: monmap: 2024-03-25T18:24:47.504 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 0 2024-03-25T18:24:47.504 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:47.504 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 0.000000 2024-03-25T18:24:47.504 INFO:teuthology.orchestra.run.smithi167.stderr:created 0.000000 2024-03-25T18:24:47.504 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:47.505 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 1 2024-03-25T18:24:47.505 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:24:47.505 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:24:47.505 INFO:teuthology.orchestra.run.smithi167.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:24:47.505 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:24:47.505 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.500+0000 7fd979ab3640 1 librados: starting msgr at 2024-03-25T18:24:47.505 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.500+0000 7fd979ab3640 1 librados: starting objecter 2024-03-25T18:24:47.505 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.500+0000 7fd979ab3640 1 Processor -- start 2024-03-25T18:24:47.505 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.500+0000 7fd979ab3640 1 -- start start 2024-03-25T18:24:47.505 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.500+0000 7fd979ab3640 1 librados: setting wanted keys 2024-03-25T18:24:47.505 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.500+0000 7fd979ab3640 1 librados: calling monclient init 2024-03-25T18:24:47.505 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.500+0000 7fd979ab3640 10 monclient: init 2024-03-25T18:24:47.505 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.501+0000 7fd979ab3640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:47.505 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.501+0000 7fd979ab3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:24:47.505 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.501+0000 7fd979ab3640 10 monclient(hunting): picked mon.noname-a con 0x7fd9740a96b0 addr v1:172.21.15.155:6789/0 2024-03-25T18:24:47.506 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.501+0000 7fd979ab3640 10 monclient(hunting): picked mon.noname-b con 0x7fd9740b4700 addr v1:172.21.15.167:6789/0 2024-03-25T18:24:47.506 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.501+0000 7fd979ab3640 10 monclient(hunting): picked mon.noname-c con 0x7fd974142a20 addr v1:172.21.15.155:6790/0 2024-03-25T18:24:47.506 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.501+0000 7fd979ab3640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd97413c3c0 con 0x7fd9740a96b0 2024-03-25T18:24:47.506 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.501+0000 7fd979ab3640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9740ac100 con 0x7fd9740b4700 2024-03-25T18:24:47.506 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.501+0000 7fd979ab3640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9741d7280 con 0x7fd974142a20 2024-03-25T18:24:47.506 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.501+0000 7fd979ab3640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:47.506 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.501+0000 7fd979ab3640 10 monclient(hunting): authenticate will time out at 7517.623123s 2024-03-25T18:24:47.506 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.501+0000 7fd973fff640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fd974142a20 0x7fd9741e1d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.167:52004/0 (socket says 172.21.15.167:52004) 2024-03-25T18:24:47.506 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.501+0000 7fd973fff640 1 -- 172.21.15.167:0/683178173 learned_addr learned my addr 172.21.15.167:0/683178173 (peer_addr_for_me v1:172.21.15.167:0/0) 2024-03-25T18:24:47.506 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.502+0000 7fd95bfff640 10 client.?.objecter ms_handle_connect 0x7fd9740b4700 2024-03-25T18:24:47.506 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.502+0000 7fd95bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:47.506 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.502+0000 7fd95bfff640 10 client.?.objecter ms_handle_connect 0x7fd974142a20 2024-03-25T18:24:47.506 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.502+0000 7fd95bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:47.506 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.502+0000 7fd95bfff640 10 client.?.objecter ms_handle_connect 0x7fd9740a96b0 2024-03-25T18:24:47.506 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.502+0000 7fd95bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:47.507 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.504+0000 7fd95bfff640 1 -- 172.21.15.167:0/683178173 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4135268795 0 0) 0x7fd9740ac100 con 0x7fd9740b4700 2024-03-25T18:24:47.508 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.504+0000 7fd95bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:47.508 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.504+0000 7fd95bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:47.508 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.504+0000 7fd95bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:47.508 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.504+0000 7fd95bfff640 10 monclient(hunting): my global_id is 4138 2024-03-25T18:24:47.508 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.504+0000 7fd95bfff640 1 -- 172.21.15.167:0/683178173 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd960004eb0 con 0x7fd9740b4700 2024-03-25T18:24:47.508 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.504+0000 7fd95bfff640 1 -- 172.21.15.167:0/683178173 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1710334740 0 0) 0x7fd97413c3c0 con 0x7fd9740a96b0 2024-03-25T18:24:47.508 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.504+0000 7fd95bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:47.508 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.504+0000 7fd95bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:47.508 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.504+0000 7fd95bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:47.508 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.504+0000 7fd95bfff640 10 monclient(hunting): my global_id is 4149 2024-03-25T18:24:47.508 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.504+0000 7fd95bfff640 1 -- 172.21.15.167:0/683178173 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9740ac100 con 0x7fd9740a96b0 2024-03-25T18:24:47.508 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.504+0000 7fd95bfff640 1 -- 172.21.15.167:0/683178173 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3757758471 0 0) 0x7fd9741d7280 con 0x7fd974142a20 2024-03-25T18:24:47.508 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.504+0000 7fd95bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:47.508 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.504+0000 7fd95bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:47.508 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.504+0000 7fd95bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:47.509 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.504+0000 7fd95bfff640 10 monclient(hunting): my global_id is 4145 2024-03-25T18:24:47.509 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.504+0000 7fd95bfff640 1 -- 172.21.15.167:0/683178173 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd97413c3c0 con 0x7fd974142a20 2024-03-25T18:24:47.509 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.508+0000 7fd95bfff640 1 -- 172.21.15.167:0/683178173 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3245581618 0 0) 0x7fd960004eb0 con 0x7fd9740b4700 2024-03-25T18:24:47.509 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.508+0000 7fd95bfff640 1 -- 172.21.15.167:0/683178173 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd9741d7280 con 0x7fd9740b4700 2024-03-25T18:24:47.509 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.508+0000 7fd95bfff640 1 -- 172.21.15.167:0/683178173 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fd968001950 con 0x7fd9740b4700 2024-03-25T18:24:47.509 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.508+0000 7fd95bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:47.510 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.508+0000 7fd95bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:47.510 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.508+0000 7fd95bfff640 10 monclient(hunting): dump: 2024-03-25T18:24:47.510 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-03-25T18:24:47.510 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:47.510 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:47.510 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:47.510 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:47.510 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 3 2024-03-25T18:24:47.510 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:47.510 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:47.510 INFO:teuthology.orchestra.run.smithi167.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:47.510 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:24:47.510 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.508+0000 7fd95bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:47.512 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.509+0000 7fd95bfff640 1 -- 172.21.15.167:0/683178173 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2665736945 0 0) 0x7fd9740ac100 con 0x7fd9740a96b0 2024-03-25T18:24:47.512 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.509+0000 7fd95bfff640 1 -- 172.21.15.167:0/683178173 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd960004eb0 con 0x7fd9740a96b0 2024-03-25T18:24:47.512 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.509+0000 7fd95bfff640 1 -- 172.21.15.167:0/683178173 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fd964001870 con 0x7fd9740a96b0 2024-03-25T18:24:47.513 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.509+0000 7fd95bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:47.513 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.509+0000 7fd95bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:47.513 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.509+0000 7fd95bfff640 10 monclient(hunting): dump: 2024-03-25T18:24:47.513 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-03-25T18:24:47.513 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:47.513 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:47.513 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:47.513 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:47.513 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 3 2024-03-25T18:24:47.513 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:47.513 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:47.513 INFO:teuthology.orchestra.run.smithi167.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:47.513 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:24:47.513 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.510+0000 7fd95bfff640 1 -- 172.21.15.167:0/683178173 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 371493812 0 0) 0x7fd97413c3c0 con 0x7fd974142a20 2024-03-25T18:24:47.513 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.510+0000 7fd95bfff640 1 -- 172.21.15.167:0/683178173 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd9740ac100 con 0x7fd974142a20 2024-03-25T18:24:47.513 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.510+0000 7fd95bfff640 1 -- 172.21.15.167:0/683178173 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fd96c001f20 con 0x7fd974142a20 2024-03-25T18:24:47.514 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.510+0000 7fd95bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:47.514 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.510+0000 7fd95bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:47.514 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.510+0000 7fd95bfff640 10 monclient(hunting): dump: 2024-03-25T18:24:47.514 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-03-25T18:24:47.514 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:47.514 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:47.514 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:47.514 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:47.514 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 3 2024-03-25T18:24:47.514 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:47.514 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:47.514 INFO:teuthology.orchestra.run.smithi167.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:47.514 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:24:47.514 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.511+0000 7fd95bfff640 1 -- 172.21.15.167:0/683178173 <== mon.1 v1:172.21.15.167:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4149308668 0 0) 0x7fd9741d7280 con 0x7fd9740b4700 2024-03-25T18:24:47.514 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.511+0000 7fd95bfff640 1 -- 172.21.15.167:0/683178173 >> v1:172.21.15.155:6790/0 conn(0x7fd974142a20 legacy=0x7fd9741e1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:47.514 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.511+0000 7fd95bfff640 1 -- 172.21.15.167:0/683178173 >> v1:172.21.15.155:6789/0 conn(0x7fd9740a96b0 legacy=0x7fd97413bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:47.515 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.511+0000 7fd95bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:24:47.515 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.511+0000 7fd95bfff640 1 monclient: found mon.b 2024-03-25T18:24:47.515 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.511+0000 7fd95bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:47.515 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.511+0000 7fd95bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:24:47.515 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.511+0000 7fd95bfff640 1 -- 172.21.15.167:0/683178173 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9741d8150 con 0x7fd9740b4700 2024-03-25T18:24:47.515 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.511+0000 7fd95bfff640 10 monclient: _finish_auth 0 2024-03-25T18:24:47.515 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.511+0000 7fd95bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:24:47.515 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.511+0000 7fd95bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:47.515 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.511+0000 7fd979ab3640 5 monclient: authenticate success, global_id 4138 2024-03-25T18:24:47.515 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.511+0000 7fd979ab3640 10 monclient: _renew_subs 2024-03-25T18:24:47.515 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.511+0000 7fd979ab3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:24:47.515 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.511+0000 7fd979ab3640 1 -- 172.21.15.167:0/683178173 --> v1:172.21.15.167:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9741d7520 con 0x7fd9740b4700 2024-03-25T18:24:47.515 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.511+0000 7fd979ab3640 10 client.4138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:47.515 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.511+0000 7fd979ab3640 10 monclient: _renew_subs 2024-03-25T18:24:47.515 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.511+0000 7fd979ab3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:24:47.515 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.511+0000 7fd979ab3640 1 -- 172.21.15.167:0/683178173 --> v1:172.21.15.167:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd97413b1d0 con 0x7fd9740b4700 2024-03-25T18:24:47.516 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.511+0000 7fd979ab3640 1 librados: init done 2024-03-25T18:24:47.516 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.512+0000 7fd979ab3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:47.516 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.512+0000 7fd979ab3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:47.516 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.512+0000 7fd979ab3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:24:47.516 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.512+0000 7fd979ab3640 1 -- 172.21.15.167:0/683178173 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd97404f730 con 0x7fd9740b4700 2024-03-25T18:24:47.568 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-25T18:24:47.566+0000 7f91e582f040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T18:24:47.615 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.614+0000 7fd95bfff640 1 -- 172.21.15.167:0/683178173 <== mon.1 v1:172.21.15.167:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd968003610 con 0x7fd9740b4700 2024-03-25T18:24:47.615 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.614+0000 7fd95bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:47.615 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.614+0000 7fd95bfff640 1 -- 172.21.15.167:0/683178173 <== mon.1 v1:172.21.15.167:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fd968001d80 con 0x7fd9740b4700 2024-03-25T18:24:47.615 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.614+0000 7fd95bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:47.615 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.614+0000 7fd95bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:47.615 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.614+0000 7fd95bfff640 10 monclient: dump: 2024-03-25T18:24:47.615 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-03-25T18:24:47.616 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:47.616 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:47.616 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:47.616 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:47.616 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 3 2024-03-25T18:24:47.616 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:47.616 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:47.616 INFO:teuthology.orchestra.run.smithi167.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:47.616 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:24:47.616 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.614+0000 7fd95bfff640 1 -- 172.21.15.167:0/683178173 <== mon.1 v1:172.21.15.167:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fd968003cf0 con 0x7fd9740b4700 2024-03-25T18:24:47.616 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.614+0000 7fd971ffb640 4 set_mon_vals no callback set 2024-03-25T18:24:47.616 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.614+0000 7fd95bfff640 1 -- 172.21.15.167:0/683178173 <== mon.1 v1:172.21.15.167:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 28265961 0 0) 0x7fd968005490 con 0x7fd9740b4700 2024-03-25T18:24:47.617 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.614+0000 7fd95bfff640 10 client.4138.objecter ms_dispatch 0x7fd974000b90 osd_map(7..7 src has 1..7) v4 2024-03-25T18:24:47.617 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.614+0000 7fd95bfff640 3 client.4138.objecter handle_osd_map got epochs [7,7] > 0 2024-03-25T18:24:47.617 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.614+0000 7fd95bfff640 3 client.4138.objecter handle_osd_map decoding full epoch 7 2024-03-25T18:24:47.617 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.614+0000 7fd95bfff640 20 client.4138.objecter dump_active .. 0 homeless 2024-03-25T18:24:47.617 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.614+0000 7fd95bfff640 1 -- 172.21.15.167:0/683178173 <== mon.1 v1:172.21.15.167:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fd968005810 con 0x7fd9740b4700 2024-03-25T18:24:47.617 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.614+0000 7fd95bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:47.617 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.614+0000 7fd95bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:47.687 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.686+0000 7fd979ab3640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "bfddfbda-7f87-471d-a105-bc2766a64f83", "id": 5}] 2024-03-25T18:24:47.687 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.686+0000 7fd979ab3640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "bfddfbda-7f87-471d-a105-bc2766a64f83", "id": 5}] 2024-03-25T18:24:47.687 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.686+0000 7fd979ab3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:24:47.687 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.686+0000 7fd979ab3640 1 -- 172.21.15.167:0/683178173 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "osd new", "uuid": "bfddfbda-7f87-471d-a105-bc2766a64f83", "id": 5} v 0) v1 -- 0x7fd974069f30 con 0x7fd9740b4700 2024-03-25T18:24:47.745 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.744+0000 7fd95bfff640 1 -- 172.21.15.167:0/683178173 <== mon.1 v1:172.21.15.167:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "bfddfbda-7f87-471d-a105-bc2766a64f83", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 1707140236 0 1988412176) 0x7fd968004f80 con 0x7fd9740b4700 2024-03-25T18:24:47.745 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.744+0000 7fd95bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "bfddfbda-7f87-471d-a105-bc2766a64f83", "id": 5}] 2024-03-25T18:24:47.745 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.744+0000 7fd95bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:47.745 INFO:teuthology.orchestra.run.smithi167.stdout:5 2024-03-25T18:24:47.746 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.745+0000 7fd979ab3640 10 librados: watch_flush enter 2024-03-25T18:24:47.747 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.745+0000 7fd979ab3640 10 librados: watch_flush exit 2024-03-25T18:24:47.747 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.745+0000 7fd979ab3640 20 client.4138.objecter shutdown clearing up homeless session... 2024-03-25T18:24:47.747 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.745+0000 7fd979ab3640 10 client.4138.objecter successfully canceled tick 2024-03-25T18:24:47.747 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.745+0000 7fd979ab3640 10 monclient: shutdown 2024-03-25T18:24:47.747 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.745+0000 7fd979ab3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:47.747 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.745+0000 7fd979ab3640 1 -- 172.21.15.167:0/683178173 >> v1:172.21.15.167:6789/0 conn(0x7fd9740b4700 legacy=0x7fd9741d7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:47.747 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.745+0000 7fd979ab3640 1 -- 172.21.15.167:0/683178173 shutdown_connections 2024-03-25T18:24:47.747 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.745+0000 7fd979ab3640 1 -- 172.21.15.167:0/683178173 >> 172.21.15.167:0/683178173 conn(0x7fd9740668e0 msgr2=0x7fd97413d320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:47.747 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.745+0000 7fd979ab3640 1 -- 172.21.15.167:0/683178173 shutdown_connections 2024-03-25T18:24:47.747 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.745+0000 7fd979ab3640 1 -- 172.21.15.167:0/683178173 wait complete. 2024-03-25T18:24:47.747 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.745+0000 7fd979ab3640 1 librados: shutdown 2024-03-25T18:24:47.756 DEBUG:teuthology.orchestra.run.smithi167:> sudo ceph --cluster ceph osd new fd4959b0-29a4-4797-8a9a-af1749e2524f 6 2024-03-25T18:24:47.854 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.853+0000 7f858f455640 10 monclient: get_monmap_and_config 2024-03-25T18:24:47.854 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.853+0000 7f858f455640 10 monclient: build_initial_monmap 2024-03-25T18:24:47.854 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.853+0000 7f858f455640 10 monclient: monmap: 2024-03-25T18:24:47.855 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 0 2024-03-25T18:24:47.855 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:47.855 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-03-25T18:24:47.854738+0000 2024-03-25T18:24:47.855 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-03-25T18:24:47.854738+0000 2024-03-25T18:24:47.855 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:47.855 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 3 2024-03-25T18:24:47.855 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:24:47.855 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:24:47.855 INFO:teuthology.orchestra.run.smithi167.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:24:47.855 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:24:47.856 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.854+0000 7f858f455640 1 Processor -- start 2024-03-25T18:24:47.856 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.854+0000 7f858f455640 1 -- start start 2024-03-25T18:24:47.856 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.854+0000 7f858f455640 10 monclient: init 2024-03-25T18:24:47.856 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.855+0000 7f858f455640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:47.856 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.855+0000 7f858f455640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:24:47.856 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.855+0000 7f858f455640 10 monclient(hunting): picked mon.noname-a con 0x7f85880a9920 addr v1:172.21.15.155:6789/0 2024-03-25T18:24:47.856 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.855+0000 7f858f455640 10 monclient(hunting): picked mon.noname-b con 0x7f858813fce0 addr v1:172.21.15.155:6790/0 2024-03-25T18:24:47.857 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.855+0000 7f858f455640 10 monclient(hunting): picked mon.noname-c con 0x7f8588142ee0 addr v1:172.21.15.167:6789/0 2024-03-25T18:24:47.857 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.855+0000 7f858f455640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85880aa430 con 0x7f85880a9920 2024-03-25T18:24:47.857 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.855+0000 7f858f455640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85880b27f0 con 0x7f8588142ee0 2024-03-25T18:24:47.857 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.855+0000 7f858f455640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85880b88e0 con 0x7f858813fce0 2024-03-25T18:24:47.857 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.855+0000 7f858f455640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:47.857 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.855+0000 7f858f455640 10 monclient(hunting): authenticate will time out at 7517.976971s 2024-03-25T18:24:47.857 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.855+0000 7f858d9f2640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f8588142ee0 0x7f858814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.167:46286/0 (socket says 172.21.15.167:46286) 2024-03-25T18:24:47.857 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.855+0000 7f858d9f2640 1 -- 172.21.15.167:0/287736340 learned_addr learned my addr 172.21.15.167:0/287736340 (peer_addr_for_me v1:172.21.15.167:0/0) 2024-03-25T18:24:47.858 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.858+0000 7f857ffff640 1 -- 172.21.15.167:0/287736340 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3726308918 0 0) 0x7f85880b27f0 con 0x7f8588142ee0 2024-03-25T18:24:47.858 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.858+0000 7f857ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:47.859 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.858+0000 7f857ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:47.859 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.858+0000 7f857ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:47.859 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.858+0000 7f857ffff640 10 monclient(hunting): my global_id is 4141 2024-03-25T18:24:47.859 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.858+0000 7f857ffff640 1 -- 172.21.15.167:0/287736340 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8574003520 con 0x7f8588142ee0 2024-03-25T18:24:47.860 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.858+0000 7f857ffff640 1 -- 172.21.15.167:0/287736340 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 757301185 0 0) 0x7f8580001110 con 0x7f85880a9920 2024-03-25T18:24:47.860 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.858+0000 7f857ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:47.860 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.858+0000 7f857ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:47.860 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.858+0000 7f857ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:47.860 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.858+0000 7f857ffff640 10 monclient(hunting): my global_id is 4152 2024-03-25T18:24:47.860 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.858+0000 7f857ffff640 1 -- 172.21.15.167:0/287736340 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85880b27f0 con 0x7f85880a9920 2024-03-25T18:24:47.860 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.858+0000 7f857ffff640 1 -- 172.21.15.167:0/287736340 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 312327348 0 0) 0x7f85880b88e0 con 0x7f858813fce0 2024-03-25T18:24:47.860 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.858+0000 7f857ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:47.860 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.858+0000 7f857ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:47.860 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.858+0000 7f857ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:47.860 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.858+0000 7f857ffff640 10 monclient(hunting): my global_id is 4148 2024-03-25T18:24:47.860 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.858+0000 7f857ffff640 1 -- 172.21.15.167:0/287736340 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8580001110 con 0x7f858813fce0 2024-03-25T18:24:47.862 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.861+0000 7f857ffff640 1 -- 172.21.15.167:0/287736340 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4022530591 0 0) 0x7f8574003520 con 0x7f8588142ee0 2024-03-25T18:24:47.862 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.861+0000 7f857ffff640 1 -- 172.21.15.167:0/287736340 >> v1:172.21.15.155:6790/0 conn(0x7f858813fce0 legacy=0x7f85881427d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:47.862 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.861+0000 7f857ffff640 1 -- 172.21.15.167:0/287736340 >> v1:172.21.15.155:6789/0 conn(0x7f85880a9920 legacy=0x7f858813f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:47.862 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.861+0000 7f857ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:24:47.862 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.861+0000 7f857ffff640 1 monclient: found mon.noname-c 2024-03-25T18:24:47.862 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.861+0000 7f857ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:47.862 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.861+0000 7f857ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.167:6789/0 2024-03-25T18:24:47.862 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.861+0000 7f857ffff640 1 -- 172.21.15.167:0/287736340 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85880b45e0 con 0x7f8588142ee0 2024-03-25T18:24:47.863 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.861+0000 7f857ffff640 10 monclient: _finish_auth 0 2024-03-25T18:24:47.863 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.861+0000 7f857ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:24:47.863 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.861+0000 7f857ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:47.863 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.861+0000 7f857ffff640 1 -- 172.21.15.167:0/287736340 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f8584001a10 con 0x7f8588142ee0 2024-03-25T18:24:47.863 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.861+0000 7f858f455640 5 monclient: authenticate success, global_id 4141 2024-03-25T18:24:47.863 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.861+0000 7f858f455640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:47.863 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.861+0000 7f857ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:47.863 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.861+0000 7f857ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:47.863 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.861+0000 7f857ffff640 10 monclient: dump: 2024-03-25T18:24:47.863 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-03-25T18:24:47.863 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:47.863 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:47.863 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:47.863 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:47.863 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 3 2024-03-25T18:24:47.863 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:47.864 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:47.864 INFO:teuthology.orchestra.run.smithi167.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:47.864 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:24:47.864 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.861+0000 7f858f455640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:47.866 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.862+0000 7f857ffff640 1 -- 172.21.15.167:0/287736340 <== mon.1 v1:172.21.15.167:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8584001b70 con 0x7f8588142ee0 2024-03-25T18:24:47.867 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.862+0000 7f857ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:47.867 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.862+0000 7f857ffff640 1 -- 172.21.15.167:0/287736340 <== mon.1 v1:172.21.15.167:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f8584001f80 con 0x7f8588142ee0 2024-03-25T18:24:47.867 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.862+0000 7f857ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:47.867 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.862+0000 7f857ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:47.867 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.862+0000 7f857ffff640 10 monclient: dump: 2024-03-25T18:24:47.867 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-03-25T18:24:47.867 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:47.867 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:47.867 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:47.867 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:47.867 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 3 2024-03-25T18:24:47.867 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:47.867 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:47.867 INFO:teuthology.orchestra.run.smithi167.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:47.867 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:24:47.867 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.862+0000 7f858f455640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:47.868 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.862+0000 7f858f455640 4 set_mon_vals no callback set 2024-03-25T18:24:47.868 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.863+0000 7f858f455640 10 monclient: shutdown 2024-03-25T18:24:47.868 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.863+0000 7f858f455640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:47.868 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.863+0000 7f858f455640 1 -- 172.21.15.167:0/287736340 >> v1:172.21.15.167:6789/0 conn(0x7f8588142ee0 legacy=0x7f858814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:47.868 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.863+0000 7f858f455640 1 -- 172.21.15.167:0/287736340 shutdown_connections 2024-03-25T18:24:47.868 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.863+0000 7f858f455640 1 -- 172.21.15.167:0/287736340 >> 172.21.15.167:0/287736340 conn(0x7f85880b7df0 msgr2=0x7f85880b81d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:47.868 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.863+0000 7f858f455640 1 -- 172.21.15.167:0/287736340 shutdown_connections 2024-03-25T18:24:47.868 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.863+0000 7f858f455640 1 -- 172.21.15.167:0/287736340 wait complete. 2024-03-25T18:24:47.868 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.864+0000 7f858f455640 10 monclient: build_initial_monmap 2024-03-25T18:24:47.868 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.864+0000 7f858f455640 10 monclient: monmap: 2024-03-25T18:24:47.868 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 0 2024-03-25T18:24:47.868 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:47.868 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 0.000000 2024-03-25T18:24:47.868 INFO:teuthology.orchestra.run.smithi167.stderr:created 0.000000 2024-03-25T18:24:47.869 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:47.869 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 1 2024-03-25T18:24:47.869 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:24:47.869 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:24:47.869 INFO:teuthology.orchestra.run.smithi167.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:24:47.869 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:24:47.869 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.864+0000 7f858f455640 1 librados: starting msgr at 2024-03-25T18:24:47.869 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.864+0000 7f858f455640 1 librados: starting objecter 2024-03-25T18:24:47.869 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.864+0000 7f858f455640 1 Processor -- start 2024-03-25T18:24:47.869 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.864+0000 7f858f455640 1 -- start start 2024-03-25T18:24:47.869 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.864+0000 7f858f455640 1 librados: setting wanted keys 2024-03-25T18:24:47.869 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.864+0000 7f858f455640 1 librados: calling monclient init 2024-03-25T18:24:47.869 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.864+0000 7f858f455640 10 monclient: init 2024-03-25T18:24:47.869 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.865+0000 7f858f455640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:47.869 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.865+0000 7f858f455640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:24:47.869 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.865+0000 7f858f455640 10 monclient(hunting): picked mon.noname-c con 0x7f85880a9920 addr v1:172.21.15.155:6790/0 2024-03-25T18:24:47.869 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.865+0000 7f858f455640 10 monclient(hunting): picked mon.noname-b con 0x7f858813fce0 addr v1:172.21.15.167:6789/0 2024-03-25T18:24:47.870 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.865+0000 7f858f455640 10 monclient(hunting): picked mon.noname-a con 0x7f8588142ee0 addr v1:172.21.15.155:6789/0 2024-03-25T18:24:47.870 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.865+0000 7f858f455640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85881dab40 con 0x7f8588142ee0 2024-03-25T18:24:47.870 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.865+0000 7f858f455640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85881437b0 con 0x7f858813fce0 2024-03-25T18:24:47.870 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.865+0000 7f858f455640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85881442f0 con 0x7f85880a9920 2024-03-25T18:24:47.870 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.865+0000 7f858f455640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:47.870 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.865+0000 7f858f455640 10 monclient(hunting): authenticate will time out at 7517.987147s 2024-03-25T18:24:47.870 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.865+0000 7f858d1f1640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f85880a9920 0x7f8588145c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.167:52024/0 (socket says 172.21.15.167:52024) 2024-03-25T18:24:47.870 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.865+0000 7f858d1f1640 1 -- 172.21.15.167:0/246730882 learned_addr learned my addr 172.21.15.167:0/246730882 (peer_addr_for_me v1:172.21.15.167:0/0) 2024-03-25T18:24:47.870 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.865+0000 7f858c9f0640 1 --1- 172.21.15.167:0/246730882 >> v1:172.21.15.167:6789/0 conn(0x7f858813fce0 0x7f85881daf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.167:46300/0 (socket says 172.21.15.167:46300) 2024-03-25T18:24:47.870 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.866+0000 7f857dffb640 10 client.?.objecter ms_handle_connect 0x7f85880a9920 2024-03-25T18:24:47.870 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.866+0000 7f857dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:47.870 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.866+0000 7f857dffb640 10 client.?.objecter ms_handle_connect 0x7f858813fce0 2024-03-25T18:24:47.870 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.866+0000 7f857dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:47.870 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.866+0000 7f857dffb640 10 client.?.objecter ms_handle_connect 0x7f8588142ee0 2024-03-25T18:24:47.871 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.866+0000 7f857dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:47.871 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.867+0000 7f857dffb640 1 -- 172.21.15.167:0/246730882 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3647915908 0 0) 0x7f85881437b0 con 0x7f858813fce0 2024-03-25T18:24:47.872 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.867+0000 7f857dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:24:47.872 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.867+0000 7f857dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:47.872 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.867+0000 7f857dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:47.872 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.868+0000 7f857dffb640 10 monclient(hunting): my global_id is 4144 2024-03-25T18:24:47.872 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.868+0000 7f857dffb640 1 -- 172.21.15.167:0/246730882 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f856c004eb0 con 0x7f858813fce0 2024-03-25T18:24:47.872 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.868+0000 7f857dffb640 1 -- 172.21.15.167:0/246730882 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1070661133 0 0) 0x7f85881442f0 con 0x7f85880a9920 2024-03-25T18:24:47.872 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.868+0000 7f857dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:24:47.872 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.868+0000 7f857dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:47.872 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.868+0000 7f857dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:47.872 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.868+0000 7f857dffb640 10 monclient(hunting): my global_id is 4151 2024-03-25T18:24:47.872 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.868+0000 7f857dffb640 1 -- 172.21.15.167:0/246730882 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85881437b0 con 0x7f85880a9920 2024-03-25T18:24:47.872 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.868+0000 7f857dffb640 1 -- 172.21.15.167:0/246730882 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1000174216 0 0) 0x7f85881dab40 con 0x7f8588142ee0 2024-03-25T18:24:47.872 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.868+0000 7f857dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:24:47.872 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.868+0000 7f857dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:47.872 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.868+0000 7f857dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:47.873 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.868+0000 7f857dffb640 10 monclient(hunting): my global_id is 4155 2024-03-25T18:24:47.873 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.868+0000 7f857dffb640 1 -- 172.21.15.167:0/246730882 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85881442f0 con 0x7f8588142ee0 2024-03-25T18:24:47.873 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.872+0000 7f857dffb640 1 -- 172.21.15.167:0/246730882 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3041257897 0 0) 0x7f856c004eb0 con 0x7f858813fce0 2024-03-25T18:24:47.873 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.872+0000 7f857dffb640 1 -- 172.21.15.167:0/246730882 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f85881dab40 con 0x7f858813fce0 2024-03-25T18:24:47.873 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.872+0000 7f857dffb640 1 -- 172.21.15.167:0/246730882 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f8578001970 con 0x7f858813fce0 2024-03-25T18:24:47.873 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.872+0000 7f857dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:47.873 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.872+0000 7f857dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:47.874 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.872+0000 7f857dffb640 10 monclient(hunting): dump: 2024-03-25T18:24:47.874 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-03-25T18:24:47.874 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:47.874 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:47.874 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:47.874 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:47.874 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 3 2024-03-25T18:24:47.874 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:47.874 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:47.874 INFO:teuthology.orchestra.run.smithi167.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:47.874 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:24:47.874 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.872+0000 7f857dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:47.876 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.874+0000 7f857dffb640 1 -- 172.21.15.167:0/246730882 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3191424268 0 0) 0x7f85881437b0 con 0x7f85880a9920 2024-03-25T18:24:47.876 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.874+0000 7f857dffb640 1 -- 172.21.15.167:0/246730882 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f856c004eb0 con 0x7f85880a9920 2024-03-25T18:24:47.876 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.874+0000 7f857dffb640 1 -- 172.21.15.167:0/246730882 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f8580001a20 con 0x7f85880a9920 2024-03-25T18:24:47.876 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.874+0000 7f857dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:47.876 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.874+0000 7f857dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:47.876 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.874+0000 7f857dffb640 10 monclient(hunting): dump: 2024-03-25T18:24:47.876 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-03-25T18:24:47.876 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:47.876 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:47.876 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:47.876 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:47.876 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 3 2024-03-25T18:24:47.876 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:47.877 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:47.877 INFO:teuthology.orchestra.run.smithi167.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:47.877 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:24:47.877 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.874+0000 7f857dffb640 1 -- 172.21.15.167:0/246730882 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3202016140 0 0) 0x7f85881442f0 con 0x7f8588142ee0 2024-03-25T18:24:47.877 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.874+0000 7f857dffb640 1 -- 172.21.15.167:0/246730882 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f85881437b0 con 0x7f8588142ee0 2024-03-25T18:24:47.877 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.874+0000 7f857dffb640 1 -- 172.21.15.167:0/246730882 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f8584001e80 con 0x7f8588142ee0 2024-03-25T18:24:47.877 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.874+0000 7f857dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:47.877 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.874+0000 7f857dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:47.877 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.874+0000 7f857dffb640 10 monclient(hunting): dump: 2024-03-25T18:24:47.877 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-03-25T18:24:47.877 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:47.877 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:47.877 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:47.877 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:47.877 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 3 2024-03-25T18:24:47.877 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:47.878 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:47.878 INFO:teuthology.orchestra.run.smithi167.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:47.878 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:24:47.879 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.876+0000 7f857dffb640 1 -- 172.21.15.167:0/246730882 <== mon.1 v1:172.21.15.167:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1031102674 0 0) 0x7f85881dab40 con 0x7f858813fce0 2024-03-25T18:24:47.879 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.876+0000 7f857dffb640 1 -- 172.21.15.167:0/246730882 >> v1:172.21.15.155:6790/0 conn(0x7f85880a9920 legacy=0x7f8588145c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:47.879 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.876+0000 7f857dffb640 1 -- 172.21.15.167:0/246730882 >> v1:172.21.15.155:6789/0 conn(0x7f8588142ee0 legacy=0x7f85881e2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:47.879 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.876+0000 7f857dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:24:47.879 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.876+0000 7f857dffb640 1 monclient: found mon.b 2024-03-25T18:24:47.879 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.876+0000 7f857dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:47.880 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.876+0000 7f857dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:24:47.880 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.876+0000 7f857dffb640 1 -- 172.21.15.167:0/246730882 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8588144480 con 0x7f858813fce0 2024-03-25T18:24:47.880 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.876+0000 7f857dffb640 10 monclient: _finish_auth 0 2024-03-25T18:24:47.880 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.876+0000 7f857dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:24:47.880 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.876+0000 7f857dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:47.880 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.876+0000 7f858f455640 5 monclient: authenticate success, global_id 4144 2024-03-25T18:24:47.880 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.876+0000 7f858f455640 10 monclient: _renew_subs 2024-03-25T18:24:47.880 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.876+0000 7f858f455640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:24:47.880 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.876+0000 7f858f455640 1 -- 172.21.15.167:0/246730882 --> v1:172.21.15.167:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85880b72b0 con 0x7f858813fce0 2024-03-25T18:24:47.880 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.876+0000 7f858f455640 10 client.4144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:47.880 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.876+0000 7f858f455640 10 monclient: _renew_subs 2024-03-25T18:24:47.880 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.876+0000 7f858f455640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:24:47.880 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.876+0000 7f858f455640 1 -- 172.21.15.167:0/246730882 --> v1:172.21.15.167:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f858813c770 con 0x7f858813fce0 2024-03-25T18:24:47.880 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.876+0000 7f858f455640 1 librados: init done 2024-03-25T18:24:47.880 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.877+0000 7f858f455640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:47.880 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.877+0000 7f858f455640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:47.881 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.877+0000 7f858f455640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:24:47.881 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.877+0000 7f858f455640 1 -- 172.21.15.167:0/246730882 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8588143980 con 0x7f858813fce0 2024-03-25T18:24:47.881 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.880+0000 7f857dffb640 1 -- 172.21.15.167:0/246730882 <== mon.1 v1:172.21.15.167:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8578003610 con 0x7f858813fce0 2024-03-25T18:24:47.882 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.880+0000 7f857dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:47.882 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.880+0000 7f857dffb640 1 -- 172.21.15.167:0/246730882 <== mon.1 v1:172.21.15.167:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f8578003960 con 0x7f858813fce0 2024-03-25T18:24:47.882 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.880+0000 7f857dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:47.882 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.880+0000 7f857dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:47.882 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.880+0000 7f857dffb640 10 monclient: dump: 2024-03-25T18:24:47.882 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-03-25T18:24:47.882 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:47.882 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:47.882 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:47.882 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:47.882 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 3 2024-03-25T18:24:47.882 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:47.882 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:47.882 INFO:teuthology.orchestra.run.smithi167.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:47.883 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:24:47.883 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.880+0000 7f857f7fe640 4 set_mon_vals no callback set 2024-03-25T18:24:47.883 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.880+0000 7f857dffb640 1 -- 172.21.15.167:0/246730882 <== mon.1 v1:172.21.15.167:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f8578003f10 con 0x7f858813fce0 2024-03-25T18:24:47.883 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.880+0000 7f857dffb640 1 -- 172.21.15.167:0/246730882 <== mon.1 v1:172.21.15.167:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 772514505 0 0) 0x7f8578006770 con 0x7f858813fce0 2024-03-25T18:24:47.883 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.880+0000 7f857dffb640 10 client.4144.objecter ms_dispatch 0x7f8588000b90 osd_map(8..8 src has 1..8) v4 2024-03-25T18:24:47.883 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.880+0000 7f857dffb640 3 client.4144.objecter handle_osd_map got epochs [8,8] > 0 2024-03-25T18:24:47.883 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.880+0000 7f857dffb640 3 client.4144.objecter handle_osd_map decoding full epoch 8 2024-03-25T18:24:47.883 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:47.881+0000 7f857dffb640 20 client.4144.objecter dump_active .. 0 homeless 2024-03-25T18:24:48.032 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.031+0000 7f857dffb640 1 -- 172.21.15.167:0/246730882 <== mon.1 v1:172.21.15.167:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f85780041c0 con 0x7f858813fce0 2024-03-25T18:24:48.032 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.031+0000 7f857dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:48.032 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.031+0000 7f857dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:48.103 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.102+0000 7f858f455640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "fd4959b0-29a4-4797-8a9a-af1749e2524f", "id": 6}] 2024-03-25T18:24:48.103 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.102+0000 7f858f455640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "fd4959b0-29a4-4797-8a9a-af1749e2524f", "id": 6}] 2024-03-25T18:24:48.103 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.102+0000 7f858f455640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:24:48.103 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.102+0000 7f858f455640 1 -- 172.21.15.167:0/246730882 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "osd new", "uuid": "fd4959b0-29a4-4797-8a9a-af1749e2524f", "id": 6} v 0) v1 -- 0x7f858813a480 con 0x7f858813fce0 2024-03-25T18:24:48.191 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.190+0000 7f857dffb640 1 -- 172.21.15.167:0/246730882 <== mon.1 v1:172.21.15.167:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "fd4959b0-29a4-4797-8a9a-af1749e2524f", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 4163154937 0 1113811849) 0x7f8578004400 con 0x7f858813fce0 2024-03-25T18:24:48.191 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.190+0000 7f857dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "fd4959b0-29a4-4797-8a9a-af1749e2524f", "id": 6}] 2024-03-25T18:24:48.191 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.190+0000 7f857dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:48.191 INFO:teuthology.orchestra.run.smithi167.stdout:6 2024-03-25T18:24:48.192 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.190+0000 7f858f455640 10 librados: watch_flush enter 2024-03-25T18:24:48.192 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.190+0000 7f858f455640 10 librados: watch_flush exit 2024-03-25T18:24:48.192 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.191+0000 7f858f455640 20 client.4144.objecter shutdown clearing up homeless session... 2024-03-25T18:24:48.192 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.191+0000 7f858f455640 10 client.4144.objecter successfully canceled tick 2024-03-25T18:24:48.192 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.191+0000 7f858f455640 10 monclient: shutdown 2024-03-25T18:24:48.192 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.191+0000 7f858f455640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:48.193 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.191+0000 7f858f455640 1 -- 172.21.15.167:0/246730882 >> v1:172.21.15.167:6789/0 conn(0x7f858813fce0 legacy=0x7f85881daf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:48.193 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.191+0000 7f858f455640 1 -- 172.21.15.167:0/246730882 shutdown_connections 2024-03-25T18:24:48.193 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.191+0000 7f858f455640 1 -- 172.21.15.167:0/246730882 >> 172.21.15.167:0/246730882 conn(0x7f85880b7df0 msgr2=0x7f85880b3500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:48.193 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.191+0000 7f858f455640 1 -- 172.21.15.167:0/246730882 shutdown_connections 2024-03-25T18:24:48.193 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.191+0000 7f858f455640 1 -- 172.21.15.167:0/246730882 wait complete. 2024-03-25T18:24:48.193 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.191+0000 7f858f455640 1 librados: shutdown 2024-03-25T18:24:48.203 DEBUG:teuthology.orchestra.run.smithi167:> sudo ceph --cluster ceph osd new 5c3d7044-9a4b-42ee-941e-90eb460fd371 7 2024-03-25T18:24:48.208 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-03-25T18:24:48.207+0000 7f1820e2b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T18:24:48.304 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.302+0000 7f8a5ea60640 10 monclient: get_monmap_and_config 2024-03-25T18:24:48.304 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.302+0000 7f8a5ea60640 10 monclient: build_initial_monmap 2024-03-25T18:24:48.304 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.302+0000 7f8a5ea60640 10 monclient: monmap: 2024-03-25T18:24:48.304 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 0 2024-03-25T18:24:48.304 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:48.304 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-03-25T18:24:48.303394+0000 2024-03-25T18:24:48.304 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-03-25T18:24:48.303394+0000 2024-03-25T18:24:48.304 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:48.304 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 3 2024-03-25T18:24:48.304 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:24:48.304 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:24:48.304 INFO:teuthology.orchestra.run.smithi167.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:24:48.304 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:24:48.304 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.303+0000 7f8a5ea60640 1 Processor -- start 2024-03-25T18:24:48.305 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.303+0000 7f8a5ea60640 1 -- start start 2024-03-25T18:24:48.305 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.303+0000 7f8a5ea60640 10 monclient: init 2024-03-25T18:24:48.306 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.303+0000 7f8a5ea60640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:48.306 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.303+0000 7f8a5ea60640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:24:48.306 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.303+0000 7f8a5ea60640 10 monclient(hunting): picked mon.noname-b con 0x7f8a580b4700 addr v1:172.21.15.155:6790/0 2024-03-25T18:24:48.306 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.303+0000 7f8a5ea60640 10 monclient(hunting): picked mon.noname-a con 0x7f8a58142a20 addr v1:172.21.15.155:6789/0 2024-03-25T18:24:48.306 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.303+0000 7f8a5ea60640 10 monclient(hunting): picked mon.noname-c con 0x7f8a580a96b0 addr v1:172.21.15.167:6789/0 2024-03-25T18:24:48.306 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.303+0000 7f8a5ea60640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a580a9d60 con 0x7f8a58142a20 2024-03-25T18:24:48.306 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.303+0000 7f8a5ea60640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a5813ff10 con 0x7f8a580a96b0 2024-03-25T18:24:48.306 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.303+0000 7f8a5ea60640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a580aa1f0 con 0x7f8a580b4700 2024-03-25T18:24:48.306 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.303+0000 7f8a5ea60640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:48.306 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.303+0000 7f8a5ea60640 10 monclient(hunting): authenticate will time out at 7518.425593s 2024-03-25T18:24:48.306 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.304+0000 7f8a577fe640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f8a58142a20 0x7f8a5813f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.167:38574/0 (socket says 172.21.15.167:38574) 2024-03-25T18:24:48.307 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.304+0000 7f8a577fe640 1 -- 172.21.15.167:0/2436402915 learned_addr learned my addr 172.21.15.167:0/2436402915 (peer_addr_for_me v1:172.21.15.167:0/0) 2024-03-25T18:24:48.307 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.304+0000 7f8a57fff640 1 --1- 172.21.15.167:0/2436402915 >> v1:172.21.15.155:6790/0 conn(0x7f8a580b4700 0x7f8a58142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.167:52032/0 (socket says 172.21.15.167:52032) 2024-03-25T18:24:48.307 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.306+0000 7f8a56ffd640 1 -- 172.21.15.167:0/2436402915 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3256153801 0 0) 0x7f8a580aa1f0 con 0x7f8a580b4700 2024-03-25T18:24:48.307 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.306+0000 7f8a56ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:48.307 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.306+0000 7f8a56ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:48.308 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.306+0000 7f8a56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:48.308 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.306+0000 7f8a56ffd640 10 monclient(hunting): my global_id is 4154 2024-03-25T18:24:48.308 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.306+0000 7f8a56ffd640 1 -- 172.21.15.167:0/2436402915 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a40003520 con 0x7f8a580b4700 2024-03-25T18:24:48.308 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.306+0000 7f8a56ffd640 1 -- 172.21.15.167:0/2436402915 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 576807722 0 0) 0x7f8a580a9d60 con 0x7f8a58142a20 2024-03-25T18:24:48.308 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.306+0000 7f8a56ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:48.308 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.306+0000 7f8a56ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:48.308 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.306+0000 7f8a56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:48.308 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.306+0000 7f8a56ffd640 10 monclient(hunting): my global_id is 4158 2024-03-25T18:24:48.308 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.306+0000 7f8a56ffd640 1 -- 172.21.15.167:0/2436402915 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a580aa1f0 con 0x7f8a58142a20 2024-03-25T18:24:48.309 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.307+0000 7f8a56ffd640 1 -- 172.21.15.167:0/2436402915 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3856203362 0 0) 0x7f8a5813ff10 con 0x7f8a580a96b0 2024-03-25T18:24:48.309 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.307+0000 7f8a56ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:48.309 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.307+0000 7f8a56ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:48.309 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.307+0000 7f8a56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:48.309 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.307+0000 7f8a56ffd640 10 monclient(hunting): my global_id is 4147 2024-03-25T18:24:48.309 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.307+0000 7f8a56ffd640 1 -- 172.21.15.167:0/2436402915 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a580a9d60 con 0x7f8a580a96b0 2024-03-25T18:24:48.311 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.310+0000 7f8a56ffd640 1 -- 172.21.15.167:0/2436402915 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3963563150 0 0) 0x7f8a40003520 con 0x7f8a580b4700 2024-03-25T18:24:48.311 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.310+0000 7f8a56ffd640 1 -- 172.21.15.167:0/2436402915 >> v1:172.21.15.167:6789/0 conn(0x7f8a580a96b0 legacy=0x7f8a5813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:48.311 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.310+0000 7f8a56ffd640 1 -- 172.21.15.167:0/2436402915 >> v1:172.21.15.155:6789/0 conn(0x7f8a58142a20 legacy=0x7f8a5813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:48.311 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.310+0000 7f8a56ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:24:48.312 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.310+0000 7f8a56ffd640 1 monclient: found mon.noname-b 2024-03-25T18:24:48.312 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.310+0000 7f8a56ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:48.312 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.310+0000 7f8a56ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-25T18:24:48.312 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.310+0000 7f8a56ffd640 1 -- 172.21.15.167:0/2436402915 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a580b4ae0 con 0x7f8a580b4700 2024-03-25T18:24:48.312 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.310+0000 7f8a56ffd640 10 monclient: _finish_auth 0 2024-03-25T18:24:48.312 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.310+0000 7f8a56ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:24:48.312 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.310+0000 7f8a56ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:48.312 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.310+0000 7f8a5ea60640 5 monclient: authenticate success, global_id 4154 2024-03-25T18:24:48.312 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.310+0000 7f8a5ea60640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:48.313 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.311+0000 7f8a56ffd640 1 -- 172.21.15.167:0/2436402915 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f8a48001650 con 0x7f8a580b4700 2024-03-25T18:24:48.313 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.311+0000 7f8a56ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:48.313 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.311+0000 7f8a56ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:48.313 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.311+0000 7f8a56ffd640 10 monclient: dump: 2024-03-25T18:24:48.313 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-03-25T18:24:48.313 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:48.313 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:48.313 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:48.313 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:48.313 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 3 2024-03-25T18:24:48.313 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:48.313 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:48.313 INFO:teuthology.orchestra.run.smithi167.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:48.313 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:24:48.314 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.311+0000 7f8a5ea60640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:48.314 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.313+0000 7f8a56ffd640 1 -- 172.21.15.167:0/2436402915 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8a480018e0 con 0x7f8a580b4700 2024-03-25T18:24:48.314 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.313+0000 7f8a56ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:48.314 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.313+0000 7f8a56ffd640 1 -- 172.21.15.167:0/2436402915 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f8a48001c70 con 0x7f8a580b4700 2024-03-25T18:24:48.314 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.313+0000 7f8a56ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:48.314 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.313+0000 7f8a56ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:48.314 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.313+0000 7f8a56ffd640 10 monclient: dump: 2024-03-25T18:24:48.314 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-03-25T18:24:48.314 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:48.315 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:48.315 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:48.315 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:48.315 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 3 2024-03-25T18:24:48.315 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:48.315 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:48.315 INFO:teuthology.orchestra.run.smithi167.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:48.315 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:24:48.315 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.313+0000 7f8a5ea60640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:48.315 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.313+0000 7f8a5ea60640 4 set_mon_vals no callback set 2024-03-25T18:24:48.316 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.314+0000 7f8a5ea60640 10 monclient: shutdown 2024-03-25T18:24:48.316 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.314+0000 7f8a5ea60640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:48.316 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.314+0000 7f8a5ea60640 1 -- 172.21.15.167:0/2436402915 >> v1:172.21.15.155:6790/0 conn(0x7f8a580b4700 legacy=0x7f8a58142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:48.316 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.314+0000 7f8a5ea60640 1 -- 172.21.15.167:0/2436402915 shutdown_connections 2024-03-25T18:24:48.317 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.314+0000 7f8a5ea60640 1 -- 172.21.15.167:0/2436402915 >> 172.21.15.167:0/2436402915 conn(0x7f8a580668e0 msgr2=0x7f8a58066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:48.317 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.314+0000 7f8a5ea60640 1 -- 172.21.15.167:0/2436402915 shutdown_connections 2024-03-25T18:24:48.317 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.314+0000 7f8a5ea60640 1 -- 172.21.15.167:0/2436402915 wait complete. 2024-03-25T18:24:48.317 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.315+0000 7f8a5ea60640 10 monclient: build_initial_monmap 2024-03-25T18:24:48.317 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.315+0000 7f8a5ea60640 10 monclient: monmap: 2024-03-25T18:24:48.317 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 0 2024-03-25T18:24:48.317 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:48.317 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 0.000000 2024-03-25T18:24:48.317 INFO:teuthology.orchestra.run.smithi167.stderr:created 0.000000 2024-03-25T18:24:48.317 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:48.317 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 1 2024-03-25T18:24:48.317 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:24:48.317 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:24:48.317 INFO:teuthology.orchestra.run.smithi167.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:24:48.317 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:24:48.317 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.315+0000 7f8a5ea60640 1 librados: starting msgr at 2024-03-25T18:24:48.318 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.315+0000 7f8a5ea60640 1 librados: starting objecter 2024-03-25T18:24:48.318 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.315+0000 7f8a5ea60640 1 Processor -- start 2024-03-25T18:24:48.318 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.316+0000 7f8a5ea60640 1 -- start start 2024-03-25T18:24:48.318 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.316+0000 7f8a5ea60640 1 librados: setting wanted keys 2024-03-25T18:24:48.318 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.316+0000 7f8a5ea60640 1 librados: calling monclient init 2024-03-25T18:24:48.318 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.316+0000 7f8a5ea60640 10 monclient: init 2024-03-25T18:24:48.319 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.316+0000 7f8a5ea60640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:48.320 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.316+0000 7f8a5ea60640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:24:48.320 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.316+0000 7f8a5ea60640 10 monclient(hunting): picked mon.noname-a con 0x7f8a580a96b0 addr v1:172.21.15.155:6789/0 2024-03-25T18:24:48.320 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.316+0000 7f8a5ea60640 10 monclient(hunting): picked mon.noname-c con 0x7f8a580b4700 addr v1:172.21.15.155:6790/0 2024-03-25T18:24:48.320 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.316+0000 7f8a5ea60640 10 monclient(hunting): picked mon.noname-b con 0x7f8a58142a20 addr v1:172.21.15.167:6789/0 2024-03-25T18:24:48.320 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.316+0000 7f8a5ea60640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a58139600 con 0x7f8a580a96b0 2024-03-25T18:24:48.320 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.316+0000 7f8a5ea60640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a58139850 con 0x7f8a58142a20 2024-03-25T18:24:48.320 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.316+0000 7f8a5ea60640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a58139b30 con 0x7f8a580b4700 2024-03-25T18:24:48.320 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.316+0000 7f8a5ea60640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:48.320 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.316+0000 7f8a5ea60640 10 monclient(hunting): authenticate will time out at 7518.438722s 2024-03-25T18:24:48.320 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.317+0000 7f8a577fe640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f8a580b4700 0x7f8a58139f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.167:52046/0 (socket says 172.21.15.167:52046) 2024-03-25T18:24:48.320 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.317+0000 7f8a577fe640 1 -- 172.21.15.167:0/2735539003 learned_addr learned my addr 172.21.15.167:0/2735539003 (peer_addr_for_me v1:172.21.15.167:0/0) 2024-03-25T18:24:48.320 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.317+0000 7f8a54ff9640 10 client.?.objecter ms_handle_connect 0x7f8a580b4700 2024-03-25T18:24:48.320 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.317+0000 7f8a54ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:48.321 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.317+0000 7f8a54ff9640 10 client.?.objecter ms_handle_connect 0x7f8a580a96b0 2024-03-25T18:24:48.321 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.317+0000 7f8a54ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:48.321 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.317+0000 7f8a54ff9640 10 client.?.objecter ms_handle_connect 0x7f8a58142a20 2024-03-25T18:24:48.321 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.317+0000 7f8a54ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:48.322 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.319+0000 7f8a54ff9640 1 -- 172.21.15.167:0/2735539003 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3343109638 0 0) 0x7f8a58139850 con 0x7f8a58142a20 2024-03-25T18:24:48.322 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.319+0000 7f8a54ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:24:48.322 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.319+0000 7f8a54ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:48.322 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.319+0000 7f8a54ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:48.322 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.319+0000 7f8a54ff9640 10 monclient(hunting): my global_id is 4150 2024-03-25T18:24:48.322 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.319+0000 7f8a54ff9640 1 -- 172.21.15.167:0/2735539003 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a44004eb0 con 0x7f8a58142a20 2024-03-25T18:24:48.322 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.319+0000 7f8a54ff9640 1 -- 172.21.15.167:0/2735539003 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2571581521 0 0) 0x7f8a58139b30 con 0x7f8a580b4700 2024-03-25T18:24:48.322 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.319+0000 7f8a54ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:24:48.322 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.319+0000 7f8a54ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:48.322 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.319+0000 7f8a54ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:48.322 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.319+0000 7f8a54ff9640 10 monclient(hunting): my global_id is 4157 2024-03-25T18:24:48.322 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.319+0000 7f8a54ff9640 1 -- 172.21.15.167:0/2735539003 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a58139850 con 0x7f8a580b4700 2024-03-25T18:24:48.322 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.320+0000 7f8a54ff9640 1 -- 172.21.15.167:0/2735539003 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2082982977 0 0) 0x7f8a58139600 con 0x7f8a580a96b0 2024-03-25T18:24:48.322 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.320+0000 7f8a54ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:24:48.323 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.320+0000 7f8a54ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:48.323 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.320+0000 7f8a54ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:48.323 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.320+0000 7f8a54ff9640 10 monclient(hunting): my global_id is 4161 2024-03-25T18:24:48.323 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.320+0000 7f8a54ff9640 1 -- 172.21.15.167:0/2735539003 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a58139b30 con 0x7f8a580a96b0 2024-03-25T18:24:48.326 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.325+0000 7f8a54ff9640 1 -- 172.21.15.167:0/2735539003 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3167598808 0 0) 0x7f8a58139b30 con 0x7f8a580a96b0 2024-03-25T18:24:48.327 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.325+0000 7f8a54ff9640 1 -- 172.21.15.167:0/2735539003 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8a58139600 con 0x7f8a580a96b0 2024-03-25T18:24:48.327 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.325+0000 7f8a54ff9640 1 -- 172.21.15.167:0/2735539003 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f8a48001550 con 0x7f8a580a96b0 2024-03-25T18:24:48.327 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.325+0000 7f8a54ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:48.327 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.325+0000 7f8a54ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:24:48.327 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.325+0000 7f8a54ff9640 10 monclient(hunting): dump: 2024-03-25T18:24:48.327 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-03-25T18:24:48.327 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:48.327 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:48.327 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:48.327 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:48.327 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 3 2024-03-25T18:24:48.327 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:48.327 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:48.327 INFO:teuthology.orchestra.run.smithi167.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:48.327 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:24:48.328 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.325+0000 7f8a54ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:48.328 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.325+0000 7f8a54ff9640 1 -- 172.21.15.167:0/2735539003 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1900909109 0 0) 0x7f8a58139850 con 0x7f8a580b4700 2024-03-25T18:24:48.328 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.325+0000 7f8a54ff9640 1 -- 172.21.15.167:0/2735539003 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8a58139b30 con 0x7f8a580b4700 2024-03-25T18:24:48.328 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.325+0000 7f8a54ff9640 1 -- 172.21.15.167:0/2735539003 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f8a4c001a50 con 0x7f8a580b4700 2024-03-25T18:24:48.328 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.325+0000 7f8a54ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:48.328 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.325+0000 7f8a54ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:48.328 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.325+0000 7f8a54ff9640 10 monclient(hunting): dump: 2024-03-25T18:24:48.328 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-03-25T18:24:48.328 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:48.328 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:48.328 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:48.328 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:48.328 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 3 2024-03-25T18:24:48.328 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:48.329 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:48.329 INFO:teuthology.orchestra.run.smithi167.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:48.329 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:24:48.329 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.325+0000 7f8a54ff9640 1 -- 172.21.15.167:0/2735539003 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1827988683 0 0) 0x7f8a44004eb0 con 0x7f8a58142a20 2024-03-25T18:24:48.329 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.325+0000 7f8a54ff9640 1 -- 172.21.15.167:0/2735539003 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8a58139850 con 0x7f8a58142a20 2024-03-25T18:24:48.329 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.325+0000 7f8a54ff9640 1 -- 172.21.15.167:0/2735539003 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f8a50001b00 con 0x7f8a58142a20 2024-03-25T18:24:48.329 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.325+0000 7f8a54ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:48.329 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.325+0000 7f8a54ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:48.329 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.325+0000 7f8a54ff9640 10 monclient(hunting): dump: 2024-03-25T18:24:48.329 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-03-25T18:24:48.329 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:48.329 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:48.329 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:48.329 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:48.329 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 3 2024-03-25T18:24:48.329 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:48.329 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:48.330 INFO:teuthology.orchestra.run.smithi167.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:48.330 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:24:48.331 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.328+0000 7f8a54ff9640 1 -- 172.21.15.167:0/2735539003 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 180592520 0 0) 0x7f8a58139600 con 0x7f8a580a96b0 2024-03-25T18:24:48.331 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.328+0000 7f8a54ff9640 1 -- 172.21.15.167:0/2735539003 >> v1:172.21.15.155:6790/0 conn(0x7f8a580b4700 legacy=0x7f8a58139f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:48.331 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.328+0000 7f8a54ff9640 1 -- 172.21.15.167:0/2735539003 >> v1:172.21.15.167:6789/0 conn(0x7f8a58142a20 legacy=0x7f8a58137eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:48.331 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.328+0000 7f8a54ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:24:48.331 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.328+0000 7f8a54ff9640 1 monclient: found mon.a 2024-03-25T18:24:48.331 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.328+0000 7f8a54ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:48.331 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.328+0000 7f8a54ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:24:48.331 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.328+0000 7f8a54ff9640 1 -- 172.21.15.167:0/2735539003 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a581387d0 con 0x7f8a580a96b0 2024-03-25T18:24:48.331 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.328+0000 7f8a54ff9640 10 monclient: _finish_auth 0 2024-03-25T18:24:48.331 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.328+0000 7f8a54ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:24:48.331 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.328+0000 7f8a54ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:48.332 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.328+0000 7f8a5ea60640 5 monclient: authenticate success, global_id 4161 2024-03-25T18:24:48.332 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.328+0000 7f8a5ea60640 10 monclient: _renew_subs 2024-03-25T18:24:48.332 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.328+0000 7f8a5ea60640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:24:48.332 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.328+0000 7f8a5ea60640 1 -- 172.21.15.167:0/2735539003 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a58137460 con 0x7f8a580a96b0 2024-03-25T18:24:48.332 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.328+0000 7f8a5ea60640 10 client.4161.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:48.332 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.328+0000 7f8a5ea60640 10 monclient: _renew_subs 2024-03-25T18:24:48.332 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.328+0000 7f8a5ea60640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:24:48.332 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.328+0000 7f8a5ea60640 1 -- 172.21.15.167:0/2735539003 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a581e3480 con 0x7f8a580a96b0 2024-03-25T18:24:48.332 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.328+0000 7f8a5ea60640 1 librados: init done 2024-03-25T18:24:48.332 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.329+0000 7f8a5ea60640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:48.332 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.329+0000 7f8a5ea60640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:48.332 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.329+0000 7f8a5ea60640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:24:48.332 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.329+0000 7f8a5ea60640 1 -- 172.21.15.167:0/2735539003 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a581385c0 con 0x7f8a580a96b0 2024-03-25T18:24:48.333 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.332+0000 7f8a54ff9640 1 -- 172.21.15.167:0/2735539003 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8a48001210 con 0x7f8a580a96b0 2024-03-25T18:24:48.334 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.332+0000 7f8a54ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:48.334 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.332+0000 7f8a54ff9640 1 -- 172.21.15.167:0/2735539003 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f8a48004b30 con 0x7f8a580a96b0 2024-03-25T18:24:48.334 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.332+0000 7f8a54ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:48.334 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.332+0000 7f8a54ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:48.334 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.332+0000 7f8a54ff9640 10 monclient: dump: 2024-03-25T18:24:48.334 INFO:teuthology.orchestra.run.smithi167.stderr:epoch 1 2024-03-25T18:24:48.334 INFO:teuthology.orchestra.run.smithi167.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:48.334 INFO:teuthology.orchestra.run.smithi167.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:48.334 INFO:teuthology.orchestra.run.smithi167.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:48.334 INFO:teuthology.orchestra.run.smithi167.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:48.334 INFO:teuthology.orchestra.run.smithi167.stderr:election_strategy: 3 2024-03-25T18:24:48.334 INFO:teuthology.orchestra.run.smithi167.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:48.334 INFO:teuthology.orchestra.run.smithi167.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:48.334 INFO:teuthology.orchestra.run.smithi167.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:48.334 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:24:48.335 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.332+0000 7f8a54ff9640 1 -- 172.21.15.167:0/2735539003 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f8a48005020 con 0x7f8a580a96b0 2024-03-25T18:24:48.335 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.332+0000 7f8a54ff9640 1 -- 172.21.15.167:0/2735539003 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 2848712007 0 0) 0x7f8a48006790 con 0x7f8a580a96b0 2024-03-25T18:24:48.335 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.332+0000 7f8a567fc640 4 set_mon_vals no callback set 2024-03-25T18:24:48.335 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.332+0000 7f8a54ff9640 10 client.4161.objecter ms_dispatch 0x7f8a58000b90 osd_map(9..9 src has 1..9) v4 2024-03-25T18:24:48.335 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.332+0000 7f8a54ff9640 3 client.4161.objecter handle_osd_map got epochs [9,9] > 0 2024-03-25T18:24:48.335 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.332+0000 7f8a54ff9640 3 client.4161.objecter handle_osd_map decoding full epoch 9 2024-03-25T18:24:48.335 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.332+0000 7f8a54ff9640 20 client.4161.objecter dump_active .. 0 homeless 2024-03-25T18:24:48.429 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.428+0000 7f8a54ff9640 1 -- 172.21.15.167:0/2735539003 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f8a48006240 con 0x7f8a580a96b0 2024-03-25T18:24:48.429 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.428+0000 7f8a54ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:48.429 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.428+0000 7f8a54ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:48.501 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-03-25T18:24:48.500+0000 7f1820e2b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T18:24:48.502 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.501+0000 7f8a5ea60640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "5c3d7044-9a4b-42ee-941e-90eb460fd371", "id": 7}] 2024-03-25T18:24:48.502 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.501+0000 7f8a5ea60640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "5c3d7044-9a4b-42ee-941e-90eb460fd371", "id": 7}] 2024-03-25T18:24:48.502 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.501+0000 7f8a5ea60640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:24:48.502 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.501+0000 7f8a5ea60640 1 -- 172.21.15.167:0/2735539003 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd new", "uuid": "5c3d7044-9a4b-42ee-941e-90eb460fd371", "id": 7} v 0) v1 -- 0x7f8a5804f730 con 0x7f8a580a96b0 2024-03-25T18:24:48.530 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.529+0000 7f8a54ff9640 1 -- 172.21.15.167:0/2735539003 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5c3d7044-9a4b-42ee-941e-90eb460fd371", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 472831995 0 1371665406) 0x7f8a48006580 con 0x7f8a580a96b0 2024-03-25T18:24:48.530 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.529+0000 7f8a54ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "5c3d7044-9a4b-42ee-941e-90eb460fd371", "id": 7}] 2024-03-25T18:24:48.530 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.529+0000 7f8a54ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:48.530 INFO:teuthology.orchestra.run.smithi167.stdout:7 2024-03-25T18:24:48.531 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.529+0000 7f8a5ea60640 10 librados: watch_flush enter 2024-03-25T18:24:48.531 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.529+0000 7f8a5ea60640 10 librados: watch_flush exit 2024-03-25T18:24:48.531 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.529+0000 7f8a5ea60640 20 client.4161.objecter shutdown clearing up homeless session... 2024-03-25T18:24:48.531 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.529+0000 7f8a5ea60640 10 client.4161.objecter successfully canceled tick 2024-03-25T18:24:48.531 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.530+0000 7f8a5ea60640 10 monclient: shutdown 2024-03-25T18:24:48.532 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.530+0000 7f8a5ea60640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:48.532 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.530+0000 7f8a5ea60640 1 -- 172.21.15.167:0/2735539003 >> v1:172.21.15.155:6789/0 conn(0x7f8a580a96b0 legacy=0x7f8a58135a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:48.532 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.530+0000 7f8a5ea60640 1 -- 172.21.15.167:0/2735539003 shutdown_connections 2024-03-25T18:24:48.532 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.530+0000 7f8a5ea60640 1 -- 172.21.15.167:0/2735539003 >> 172.21.15.167:0/2735539003 conn(0x7f8a580668e0 msgr2=0x7f8a5813f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:48.532 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.530+0000 7f8a5ea60640 1 -- 172.21.15.167:0/2735539003 shutdown_connections 2024-03-25T18:24:48.532 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.530+0000 7f8a5ea60640 1 -- 172.21.15.167:0/2735539003 wait complete. 2024-03-25T18:24:48.532 INFO:teuthology.orchestra.run.smithi167.stderr:2024-03-25T18:24:48.530+0000 7f8a5ea60640 1 librados: shutdown 2024-03-25T18:24:48.542 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:24:48.542 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-25T18:24:48.542 DEBUG:teuthology.orchestra.run.smithi155:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-03-25T18:24:48.544 INFO:tasks.ceph.osd.0:Started 2024-03-25T18:24:48.545 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:24:48.545 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-25T18:24:48.545 DEBUG:teuthology.orchestra.run.smithi155:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-03-25T18:24:48.547 INFO:tasks.ceph.osd.1:Started 2024-03-25T18:24:48.547 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:24:48.547 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-25T18:24:48.547 DEBUG:teuthology.orchestra.run.smithi155:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-03-25T18:24:48.549 INFO:tasks.ceph.osd.2:Started 2024-03-25T18:24:48.549 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:24:48.549 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-25T18:24:48.550 DEBUG:teuthology.orchestra.run.smithi155:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-03-25T18:24:48.552 INFO:tasks.ceph.osd.3:Started 2024-03-25T18:24:48.552 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:24:48.552 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-25T18:24:48.552 DEBUG:teuthology.orchestra.run.smithi167:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-03-25T18:24:48.558 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-03-25T18:24:48.557+0000 7f1820e2b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T18:24:48.559 INFO:tasks.ceph.osd.4:Started 2024-03-25T18:24:48.559 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:24:48.560 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-25T18:24:48.560 DEBUG:teuthology.orchestra.run.smithi167:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-03-25T18:24:48.562 INFO:tasks.ceph.osd.5:Started 2024-03-25T18:24:48.562 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:24:48.562 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-25T18:24:48.562 DEBUG:teuthology.orchestra.run.smithi167:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-03-25T18:24:48.564 INFO:tasks.ceph.osd.6:Started 2024-03-25T18:24:48.564 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:24:48.564 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-25T18:24:48.564 DEBUG:teuthology.orchestra.run.smithi167:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-03-25T18:24:48.566 INFO:tasks.ceph.osd.7:Started 2024-03-25T18:24:48.567 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:48.708 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-03-25T18:24:48.707+0000 7f1820e2b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T18:24:48.710 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.708+0000 7fb7dd14b640 10 monclient: get_monmap_and_config 2024-03-25T18:24:48.710 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.708+0000 7fb7dd14b640 10 monclient: build_initial_monmap 2024-03-25T18:24:48.710 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.708+0000 7fb7dd14b640 10 monclient: monmap: 2024-03-25T18:24:48.710 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-03-25T18:24:48.710 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:48.711 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-03-25T18:24:48.710701+0000 2024-03-25T18:24:48.711 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-03-25T18:24:48.710701+0000 2024-03-25T18:24:48.711 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:48.711 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 3 2024-03-25T18:24:48.711 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:24:48.711 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:24:48.711 INFO:teuthology.orchestra.run.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:24:48.711 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-03-25T18:24:48.712 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.709+0000 7fb7dd14b640 1 Processor -- start 2024-03-25T18:24:48.712 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.709+0000 7fb7dd14b640 1 -- start start 2024-03-25T18:24:48.712 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.709+0000 7fb7dd14b640 10 monclient: init 2024-03-25T18:24:48.712 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.710+0000 7fb7dd14b640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:48.712 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.710+0000 7fb7dd14b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:24:48.712 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.710+0000 7fb7dd14b640 10 monclient(hunting): picked mon.noname-c con 0x7fb7d80b4700 addr v1:172.21.15.167:6789/0 2024-03-25T18:24:48.712 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.710+0000 7fb7dd14b640 10 monclient(hunting): picked mon.noname-b con 0x7fb7d8142a20 addr v1:172.21.15.155:6790/0 2024-03-25T18:24:48.713 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.710+0000 7fb7dd14b640 10 monclient(hunting): picked mon.noname-a con 0x7fb7d80a96b0 addr v1:172.21.15.155:6789/0 2024-03-25T18:24:48.713 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.710+0000 7fb7dd14b640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7d80a9d60 con 0x7fb7d80a96b0 2024-03-25T18:24:48.713 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.710+0000 7fb7dd14b640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7d813ff10 con 0x7fb7d80b4700 2024-03-25T18:24:48.714 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.711+0000 7fb7dd14b640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7d80aa1f0 con 0x7fb7d8142a20 2024-03-25T18:24:48.714 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.711+0000 7fb7dd14b640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:48.714 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.711+0000 7fb7dd14b640 10 monclient(hunting): authenticate will time out at 7535.021761s 2024-03-25T18:24:48.714 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.711+0000 7fb7d659c640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fb7d8142a20 0x7fb7d813f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:49120/0 (socket says 172.21.15.155:49120) 2024-03-25T18:24:48.714 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.711+0000 7fb7d659c640 1 -- 172.21.15.155:0/3402147373 learned_addr learned my addr 172.21.15.155:0/3402147373 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:24:48.718 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.714+0000 7fb7d5d9b640 1 -- 172.21.15.155:0/3402147373 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 823318194 0 0) 0x7fb7d80aa1f0 con 0x7fb7d8142a20 2024-03-25T18:24:48.718 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.715+0000 7fb7d5d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:24:48.718 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.715+0000 7fb7d5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:48.718 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.715+0000 7fb7d5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:48.718 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.715+0000 7fb7d5d9b640 10 monclient(hunting): my global_id is 4160 2024-03-25T18:24:48.718 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.715+0000 7fb7d5d9b640 1 -- 172.21.15.155:0/3402147373 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7c0003520 con 0x7fb7d8142a20 2024-03-25T18:24:48.718 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.715+0000 7fb7d5d9b640 1 -- 172.21.15.155:0/3402147373 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1879804521 0 0) 0x7fb7d80a9d60 con 0x7fb7d80a96b0 2024-03-25T18:24:48.719 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.715+0000 7fb7d5d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:24:48.719 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.715+0000 7fb7d5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:48.719 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.715+0000 7fb7d5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:48.719 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.715+0000 7fb7d5d9b640 10 monclient(hunting): my global_id is 4164 2024-03-25T18:24:48.719 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.715+0000 7fb7d5d9b640 1 -- 172.21.15.155:0/3402147373 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7d80aa1f0 con 0x7fb7d80a96b0 2024-03-25T18:24:48.719 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.715+0000 7fb7d5d9b640 1 -- 172.21.15.155:0/3402147373 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1867258598 0 0) 0x7fb7d813ff10 con 0x7fb7d80b4700 2024-03-25T18:24:48.719 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.715+0000 7fb7d5d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:24:48.719 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.715+0000 7fb7d5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:48.719 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.715+0000 7fb7d5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:48.719 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.716+0000 7fb7d5d9b640 10 monclient(hunting): my global_id is 4153 2024-03-25T18:24:48.719 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.716+0000 7fb7d5d9b640 1 -- 172.21.15.155:0/3402147373 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7d80a9d60 con 0x7fb7d80b4700 2024-03-25T18:24:48.723 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.720+0000 7fb7d5d9b640 1 -- 172.21.15.155:0/3402147373 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2823740531 0 0) 0x7fb7d80a9d60 con 0x7fb7d80b4700 2024-03-25T18:24:48.723 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.720+0000 7fb7d5d9b640 1 -- 172.21.15.155:0/3402147373 >> v1:172.21.15.155:6790/0 conn(0x7fb7d8142a20 legacy=0x7fb7d813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:48.724 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.720+0000 7fb7d5d9b640 1 -- 172.21.15.155:0/3402147373 >> v1:172.21.15.155:6789/0 conn(0x7fb7d80a96b0 legacy=0x7fb7d813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:48.724 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.721+0000 7fb7d5d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:24:48.724 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.721+0000 7fb7d5d9b640 1 monclient: found mon.noname-c 2024-03-25T18:24:48.724 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.721+0000 7fb7d5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:48.724 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.721+0000 7fb7d5d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.167:6789/0 2024-03-25T18:24:48.724 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.721+0000 7fb7d5d9b640 1 -- 172.21.15.155:0/3402147373 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7d80b4ae0 con 0x7fb7d80b4700 2024-03-25T18:24:48.724 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.721+0000 7fb7d5d9b640 10 monclient: _finish_auth 0 2024-03-25T18:24:48.724 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.721+0000 7fb7d5d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:24:48.724 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.721+0000 7fb7d5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:48.724 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.721+0000 7fb7d5d9b640 1 -- 172.21.15.155:0/3402147373 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fb7c8001630 con 0x7fb7d80b4700 2024-03-25T18:24:48.724 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.721+0000 7fb7d5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:48.724 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.721+0000 7fb7d5d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:48.724 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.721+0000 7fb7d5d9b640 10 monclient: dump: 2024-03-25T18:24:48.724 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-03-25T18:24:48.725 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:48.725 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:48.725 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:48.725 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:48.725 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 3 2024-03-25T18:24:48.725 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:48.725 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:48.725 INFO:teuthology.orchestra.run.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:48.725 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-03-25T18:24:48.725 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.721+0000 7fb7dd14b640 5 monclient: authenticate success, global_id 4153 2024-03-25T18:24:48.725 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.721+0000 7fb7dd14b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:48.727 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.723+0000 7fb7d5d9b640 1 -- 172.21.15.155:0/3402147373 <== mon.1 v1:172.21.15.167:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb7c8001880 con 0x7fb7d80b4700 2024-03-25T18:24:48.727 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.723+0000 7fb7d5d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:48.727 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.723+0000 7fb7d5d9b640 1 -- 172.21.15.155:0/3402147373 <== mon.1 v1:172.21.15.167:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fb7c8001c10 con 0x7fb7d80b4700 2024-03-25T18:24:48.727 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.723+0000 7fb7dd14b640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:48.727 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.723+0000 7fb7dd14b640 4 set_mon_vals no callback set 2024-03-25T18:24:48.727 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.724+0000 7fb7dd14b640 10 monclient: shutdown 2024-03-25T18:24:48.727 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.724+0000 7fb7d5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:48.727 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.724+0000 7fb7d5d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:48.727 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.724+0000 7fb7d5d9b640 10 monclient: dump: 2024-03-25T18:24:48.728 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-03-25T18:24:48.728 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:48.728 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:48.728 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:48.728 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:48.728 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 3 2024-03-25T18:24:48.728 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:48.728 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:48.728 INFO:teuthology.orchestra.run.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:48.728 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-03-25T18:24:48.728 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.724+0000 7fb7dd14b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:48.728 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.724+0000 7fb7dd14b640 1 -- 172.21.15.155:0/3402147373 >> v1:172.21.15.167:6789/0 conn(0x7fb7d80b4700 legacy=0x7fb7d8142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:48.728 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.725+0000 7fb7dd14b640 1 -- 172.21.15.155:0/3402147373 shutdown_connections 2024-03-25T18:24:48.728 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.725+0000 7fb7dd14b640 1 -- 172.21.15.155:0/3402147373 >> 172.21.15.155:0/3402147373 conn(0x7fb7d80668e0 msgr2=0x7fb7d8066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:48.729 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.725+0000 7fb7dd14b640 1 -- 172.21.15.155:0/3402147373 shutdown_connections 2024-03-25T18:24:48.729 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.725+0000 7fb7dd14b640 1 -- 172.21.15.155:0/3402147373 wait complete. 2024-03-25T18:24:48.730 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.726+0000 7fb7dd14b640 10 monclient: build_initial_monmap 2024-03-25T18:24:48.730 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.726+0000 7fb7dd14b640 10 monclient: monmap: 2024-03-25T18:24:48.730 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-03-25T18:24:48.730 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:48.730 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 0.000000 2024-03-25T18:24:48.730 INFO:teuthology.orchestra.run.smithi155.stderr:created 0.000000 2024-03-25T18:24:48.730 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:48.731 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-03-25T18:24:48.731 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:24:48.731 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:24:48.731 INFO:teuthology.orchestra.run.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:24:48.731 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-03-25T18:24:48.731 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.726+0000 7fb7dd14b640 1 librados: starting msgr at 2024-03-25T18:24:48.731 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.726+0000 7fb7dd14b640 1 librados: starting objecter 2024-03-25T18:24:48.731 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.727+0000 7fb7dd14b640 1 Processor -- start 2024-03-25T18:24:48.731 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.727+0000 7fb7dd14b640 1 -- start start 2024-03-25T18:24:48.731 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.727+0000 7fb7dd14b640 1 librados: setting wanted keys 2024-03-25T18:24:48.731 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.727+0000 7fb7dd14b640 1 librados: calling monclient init 2024-03-25T18:24:48.731 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.727+0000 7fb7dd14b640 10 monclient: init 2024-03-25T18:24:48.731 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.728+0000 7fb7dd14b640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:48.731 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.728+0000 7fb7dd14b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:24:48.731 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.728+0000 7fb7dd14b640 10 monclient(hunting): picked mon.noname-b con 0x7fb7d80a96b0 addr v1:172.21.15.167:6789/0 2024-03-25T18:24:48.733 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.729+0000 7fb7dd14b640 10 monclient(hunting): picked mon.noname-c con 0x7fb7d80b4700 addr v1:172.21.15.155:6790/0 2024-03-25T18:24:48.733 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.729+0000 7fb7dd14b640 10 monclient(hunting): picked mon.noname-a con 0x7fb7d8142a20 addr v1:172.21.15.155:6789/0 2024-03-25T18:24:48.733 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.729+0000 7fb7dd14b640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7d80ad930 con 0x7fb7d8142a20 2024-03-25T18:24:48.733 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.729+0000 7fb7d6d9d640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7fb7d80a96b0 0x7fb7d81d7940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:49042/0 (socket says 172.21.15.155:49042) 2024-03-25T18:24:48.733 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.729+0000 7fb7d6d9d640 1 -- 172.21.15.155:0/3573058296 learned_addr learned my addr 172.21.15.155:0/3573058296 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:24:48.733 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.729+0000 7fb7dd14b640 1 -- 172.21.15.155:0/3573058296 --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7d8136a50 con 0x7fb7d80a96b0 2024-03-25T18:24:48.733 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.729+0000 7fb7dd14b640 1 -- 172.21.15.155:0/3573058296 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7d81d8050 con 0x7fb7d80b4700 2024-03-25T18:24:48.733 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.729+0000 7fb7dd14b640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:48.733 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.729+0000 7fb7dd14b640 10 monclient(hunting): authenticate will time out at 7535.039977s 2024-03-25T18:24:48.733 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.729+0000 7fb7bf7fe640 10 client.?.objecter ms_handle_connect 0x7fb7d80b4700 2024-03-25T18:24:48.734 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.729+0000 7fb7bf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:48.734 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.730+0000 7fb7bf7fe640 10 client.?.objecter ms_handle_connect 0x7fb7d80a96b0 2024-03-25T18:24:48.734 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.730+0000 7fb7bf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:48.734 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-25T18:24:48.732+0000 7f91e582f040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T18:24:48.734 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.732+0000 7fb7bf7fe640 1 -- 172.21.15.155:0/3573058296 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 796851917 0 0) 0x7fb7d8136a50 con 0x7fb7d80a96b0 2024-03-25T18:24:48.734 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.732+0000 7fb7bf7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:24:48.734 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.732+0000 7fb7bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:48.734 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.732+0000 7fb7bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:48.735 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.732+0000 7fb7bf7fe640 10 monclient(hunting): my global_id is 4156 2024-03-25T18:24:48.735 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.732+0000 7fb7bf7fe640 1 -- 172.21.15.155:0/3573058296 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7c4004eb0 con 0x7fb7d80a96b0 2024-03-25T18:24:48.735 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.732+0000 7fb7bf7fe640 1 -- 172.21.15.155:0/3573058296 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3138281819 0 0) 0x7fb7d81d8050 con 0x7fb7d80b4700 2024-03-25T18:24:48.735 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.732+0000 7fb7bf7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:24:48.735 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.732+0000 7fb7bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:48.735 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.732+0000 7fb7bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:48.735 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.732+0000 7fb7bf7fe640 10 monclient(hunting): my global_id is 4163 2024-03-25T18:24:48.735 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.732+0000 7fb7bf7fe640 1 -- 172.21.15.155:0/3573058296 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7d8136a50 con 0x7fb7d80b4700 2024-03-25T18:24:48.741 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.739+0000 7fb7bf7fe640 1 -- 172.21.15.155:0/3573058296 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 142172634 0 0) 0x7fb7c4004eb0 con 0x7fb7d80a96b0 2024-03-25T18:24:48.741 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.739+0000 7fb7bf7fe640 1 -- 172.21.15.155:0/3573058296 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7d81d8050 con 0x7fb7d80a96b0 2024-03-25T18:24:48.741 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.739+0000 7fb7bf7fe640 1 -- 172.21.15.155:0/3573058296 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fb7c8001c90 con 0x7fb7d80a96b0 2024-03-25T18:24:48.741 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.739+0000 7fb7bf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:48.741 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.739+0000 7fb7bf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:48.742 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.739+0000 7fb7bf7fe640 10 monclient(hunting): dump: 2024-03-25T18:24:48.742 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-03-25T18:24:48.742 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:48.742 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:48.742 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:48.742 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:48.742 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 3 2024-03-25T18:24:48.742 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:48.742 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:48.742 INFO:teuthology.orchestra.run.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:48.742 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-03-25T18:24:48.742 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.739+0000 7fb7bf7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:48.743 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.740+0000 7fb7bf7fe640 1 -- 172.21.15.155:0/3573058296 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1381276914 0 0) 0x7fb7d8136a50 con 0x7fb7d80b4700 2024-03-25T18:24:48.743 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.740+0000 7fb7bf7fe640 1 -- 172.21.15.155:0/3573058296 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7c4004eb0 con 0x7fb7d80b4700 2024-03-25T18:24:48.743 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.740+0000 7fb7bf7fe640 1 -- 172.21.15.155:0/3573058296 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fb7cc0017b0 con 0x7fb7d80b4700 2024-03-25T18:24:48.743 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.740+0000 7fb7bf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:48.743 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.740+0000 7fb7bf7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:48.744 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.740+0000 7fb7bf7fe640 10 monclient(hunting): dump: 2024-03-25T18:24:48.744 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-03-25T18:24:48.744 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:48.744 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:48.744 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:48.744 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:48.744 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 3 2024-03-25T18:24:48.744 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:48.744 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:48.744 INFO:teuthology.orchestra.run.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:48.744 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-03-25T18:24:48.746 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.743+0000 7fb7bf7fe640 1 -- 172.21.15.155:0/3573058296 <== mon.1 v1:172.21.15.167:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3270998737 0 0) 0x7fb7d81d8050 con 0x7fb7d80a96b0 2024-03-25T18:24:48.747 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.743+0000 7fb7bf7fe640 1 -- 172.21.15.155:0/3573058296 >> v1:172.21.15.155:6790/0 conn(0x7fb7d80b4700 legacy=0x7fb7d8135c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:48.747 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.743+0000 7fb7bf7fe640 1 -- 172.21.15.155:0/3573058296 >> v1:172.21.15.155:6789/0 conn(0x7fb7d8142a20 legacy=0x7fb7d8136340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:48.747 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.743+0000 7fb7bf7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:24:48.747 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.743+0000 7fb7bf7fe640 1 monclient: found mon.b 2024-03-25T18:24:48.747 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.743+0000 7fb7bf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:48.747 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.743+0000 7fb7bf7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:24:48.747 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.743+0000 7fb7bf7fe640 1 -- 172.21.15.155:0/3573058296 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7d813cad0 con 0x7fb7d80a96b0 2024-03-25T18:24:48.747 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.743+0000 7fb7bf7fe640 10 monclient: _finish_auth 0 2024-03-25T18:24:48.747 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.743+0000 7fb7bf7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:24:48.747 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.743+0000 7fb7bf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:48.747 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.743+0000 7fb7dd14b640 5 monclient: authenticate success, global_id 4156 2024-03-25T18:24:48.747 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.743+0000 7fb7dd14b640 10 monclient: _renew_subs 2024-03-25T18:24:48.747 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.744+0000 7fb7dd14b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:24:48.747 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.744+0000 7fb7dd14b640 1 -- 172.21.15.155:0/3573058296 --> v1:172.21.15.167:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7d813cd20 con 0x7fb7d80a96b0 2024-03-25T18:24:48.748 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.744+0000 7fb7dd14b640 10 client.4156.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:48.748 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.744+0000 7fb7dd14b640 10 monclient: _renew_subs 2024-03-25T18:24:48.748 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.744+0000 7fb7dd14b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:24:48.748 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.744+0000 7fb7dd14b640 1 -- 172.21.15.155:0/3573058296 --> v1:172.21.15.167:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7d80ac740 con 0x7fb7d80a96b0 2024-03-25T18:24:48.748 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.744+0000 7fb7dd14b640 1 librados: init done 2024-03-25T18:24:48.748 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.745+0000 7fb7bd7fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:48.749 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.745+0000 7fb7bd7fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:48.749 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.746+0000 7fb7bd7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:24:48.749 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.746+0000 7fb7bd7fa640 1 -- 172.21.15.155:0/3573058296 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7d804f730 con 0x7fb7d80a96b0 2024-03-25T18:24:48.751 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.748+0000 7fb7bf7fe640 1 -- 172.21.15.155:0/3573058296 <== mon.1 v1:172.21.15.167:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb7c80038a0 con 0x7fb7d80a96b0 2024-03-25T18:24:48.751 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.748+0000 7fb7bf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:48.751 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.748+0000 7fb7bf7fe640 1 -- 172.21.15.155:0/3573058296 <== mon.1 v1:172.21.15.167:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fb7c8003c30 con 0x7fb7d80a96b0 2024-03-25T18:24:48.751 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.748+0000 7fb7bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:48.752 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.748+0000 7fb7bf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:48.752 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.748+0000 7fb7bf7fe640 10 monclient: dump: 2024-03-25T18:24:48.752 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-03-25T18:24:48.752 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:48.752 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:48.752 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:48.752 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:48.752 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 3 2024-03-25T18:24:48.752 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:48.752 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:48.752 INFO:teuthology.orchestra.run.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:48.752 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-03-25T18:24:48.752 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.748+0000 7fb7bf7fe640 1 -- 172.21.15.155:0/3573058296 <== mon.1 v1:172.21.15.167:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fb7c80041e0 con 0x7fb7d80a96b0 2024-03-25T18:24:48.752 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.748+0000 7fb7bf7fe640 1 -- 172.21.15.155:0/3573058296 <== mon.1 v1:172.21.15.167:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7fb7c80064c0 con 0x7fb7d80a96b0 2024-03-25T18:24:48.752 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.748+0000 7fb7bf7fe640 10 client.4156.objecter ms_dispatch 0x7fb7d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:24:48.752 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.748+0000 7fb7bf7fe640 3 client.4156.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:24:48.753 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.748+0000 7fb7bf7fe640 3 client.4156.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:24:48.753 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.748+0000 7fb7bf7fe640 20 client.4156.objecter dump_active .. 0 homeless 2024-03-25T18:24:48.753 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.749+0000 7fb7d559a640 4 set_mon_vals no callback set 2024-03-25T18:24:48.845 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.843+0000 7fb7bf7fe640 1 -- 172.21.15.155:0/3573058296 <== mon.1 v1:172.21.15.167:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fb7c80066f0 con 0x7fb7d80a96b0 2024-03-25T18:24:48.845 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.843+0000 7fb7bf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:48.845 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.843+0000 7fb7bf7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:48.967 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.965+0000 7fb7bd7fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:48.968 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.965+0000 7fb7bd7fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:48.968 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.965+0000 7fb7bd7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:24:48.968 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:48.965+0000 7fb7bd7fa640 1 -- 172.21.15.155:0/3573058296 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb7d80008d0 con 0x7fb7d80a96b0 2024-03-25T18:24:49.067 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:49.065+0000 7fb7bf7fe640 1 -- 172.21.15.155:0/3573058296 <== mon.1 v1:172.21.15.167:6789/0 10 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 3719313413 0 0) 0x7fb7c8005780 con 0x7fb7d80a96b0 2024-03-25T18:24:49.148 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-25T18:24:49.149 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:49.149 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:49.145+0000 7fb7bf7fe640 1 -- 172.21.15.155:0/3573058296 <== mon.1 v1:172.21.15.167:6789/0 11 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7fb7c8016050 con 0x7fb7d80a96b0 2024-03-25T18:24:49.149 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:49.145+0000 7fb7bf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:49.149 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:49.145+0000 7fb7bf7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:49.150 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:49.146+0000 7fb7dd14b640 10 librados: watch_flush enter 2024-03-25T18:24:49.150 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:49.147+0000 7fb7dd14b640 10 librados: watch_flush exit 2024-03-25T18:24:49.150 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:49.147+0000 7fb7dd14b640 20 client.4156.objecter shutdown clearing up homeless session... 2024-03-25T18:24:49.150 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:49.147+0000 7fb7dd14b640 10 client.4156.objecter successfully canceled tick 2024-03-25T18:24:49.150 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:49.147+0000 7fb7dd14b640 10 monclient: shutdown 2024-03-25T18:24:49.150 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:49.147+0000 7fb7dd14b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:49.150 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:49.147+0000 7fb7dd14b640 1 -- 172.21.15.155:0/3573058296 >> v1:172.21.15.167:6789/0 conn(0x7fb7d80a96b0 legacy=0x7fb7d81d7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:49.150 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:49.147+0000 7fb7dd14b640 1 -- 172.21.15.155:0/3573058296 shutdown_connections 2024-03-25T18:24:49.150 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:49.147+0000 7fb7dd14b640 1 -- 172.21.15.155:0/3573058296 >> 172.21.15.155:0/3573058296 conn(0x7fb7d80668e0 msgr2=0x7fb7d80b8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:49.150 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:49.147+0000 7fb7dd14b640 1 -- 172.21.15.155:0/3573058296 shutdown_connections 2024-03-25T18:24:49.150 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:49.147+0000 7fb7dd14b640 1 -- 172.21.15.155:0/3573058296 wait complete. 2024-03-25T18:24:49.150 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-25T18:24:49.147+0000 7fb7dd14b640 1 librados: shutdown 2024-03-25T18:24:49.156 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-25T18:24:49.154+0000 7f91e582f040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T18:24:49.166 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-25T18:24:49.166 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-25T18:24:49.224 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-25T18:24:49.222+0000 7f91e582f040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T18:24:49.381 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-25T18:24:49.380+0000 7f91e582f040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T18:24:49.973 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:50.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.117+0000 7f6624693640 10 monclient: get_monmap_and_config 2024-03-25T18:24:50.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.117+0000 7f6624693640 10 monclient: build_initial_monmap 2024-03-25T18:24:50.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.117+0000 7f6624693640 10 monclient: monmap: 2024-03-25T18:24:50.119 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:24:50.119 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:50.119 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:24:50.118850+0000 2024-03-25T18:24:50.119 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:24:50.118850+0000 2024-03-25T18:24:50.119 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:50.119 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:24:50.119 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:24:50.119 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:24:50.119 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:24:50.120 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:24:50.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.118+0000 7f6624693640 1 Processor -- start 2024-03-25T18:24:50.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.118+0000 7f6624693640 1 -- start start 2024-03-25T18:24:50.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.118+0000 7f6624693640 10 monclient: init 2024-03-25T18:24:50.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.119+0000 7f6624693640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:50.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.119+0000 7f6624693640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:24:50.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.119+0000 7f6624693640 10 monclient(hunting): picked mon.noname-a con 0x7f661c0a98f0 addr v1:172.21.15.155:6789/0 2024-03-25T18:24:50.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.119+0000 7f6624693640 10 monclient(hunting): picked mon.noname-b con 0x7f661c13f710 addr v1:172.21.15.155:6790/0 2024-03-25T18:24:50.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.120+0000 7f6624693640 10 monclient(hunting): picked mon.noname-c con 0x7f661c1430c0 addr v1:172.21.15.167:6789/0 2024-03-25T18:24:50.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.120+0000 7f6624693640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f661c142520 con 0x7f661c0a98f0 2024-03-25T18:24:50.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.120+0000 7f6624693640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f661c0b28a0 con 0x7f661c1430c0 2024-03-25T18:24:50.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.120+0000 7f6624693640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f661c0b2a30 con 0x7f661c13f710 2024-03-25T18:24:50.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.120+0000 7f6624693640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:50.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.120+0000 7f6624693640 10 monclient(hunting): authenticate will time out at 7536.429761s 2024-03-25T18:24:50.153 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.152+0000 7f6622c30640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f661c1430c0 0x7f661c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:49044/0 (socket says 172.21.15.155:49044) 2024-03-25T18:24:50.153 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.152+0000 7f6622c30640 1 -- 172.21.15.155:0/2536370689 learned_addr learned my addr 172.21.15.155:0/2536370689 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:24:50.162 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.161+0000 7f662142d640 1 -- 172.21.15.155:0/2536370689 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3173795153 0 0) 0x7f661c0b28a0 con 0x7f661c1430c0 2024-03-25T18:24:50.162 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.161+0000 7f662142d640 10 monclient(hunting): _negotiate 2024-03-25T18:24:50.162 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.161+0000 7f662142d640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:50.163 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.161+0000 7f662142d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:50.163 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.161+0000 7f662142d640 10 monclient(hunting): my global_id is 4174 2024-03-25T18:24:50.163 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.161+0000 7f662142d640 1 -- 172.21.15.155:0/2536370689 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6608003520 con 0x7f661c1430c0 2024-03-25T18:24:50.168 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.166+0000 7f662142d640 1 -- 172.21.15.155:0/2536370689 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1841297365 0 0) 0x7f6608003520 con 0x7f661c1430c0 2024-03-25T18:24:50.168 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.166+0000 7f662142d640 1 -- 172.21.15.155:0/2536370689 >> v1:172.21.15.155:6790/0 conn(0x7f661c13f710 legacy=0x7f661c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:24:50.168 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.166+0000 7f662142d640 1 -- 172.21.15.155:0/2536370689 >> v1:172.21.15.155:6789/0 conn(0x7f661c0a98f0 legacy=0x7f661c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:50.168 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.166+0000 7f662142d640 10 monclient: _finish_hunting 0 2024-03-25T18:24:50.168 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.166+0000 7f662142d640 1 monclient: found mon.noname-c 2024-03-25T18:24:50.169 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.166+0000 7f662142d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:50.169 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.166+0000 7f662142d640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.167:6789/0 2024-03-25T18:24:50.169 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.166+0000 7f662142d640 1 -- 172.21.15.155:0/2536370689 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f661c0b44c0 con 0x7f661c1430c0 2024-03-25T18:24:50.169 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.166+0000 7f662142d640 10 monclient: _finish_auth 0 2024-03-25T18:24:50.169 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.166+0000 7f662142d640 10 monclient: _check_auth_tickets 2024-03-25T18:24:50.169 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.166+0000 7f662142d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:50.169 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.166+0000 7f662142d640 1 -- 172.21.15.155:0/2536370689 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f6614001a10 con 0x7f661c1430c0 2024-03-25T18:24:50.169 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.166+0000 7f662142d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:50.169 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.166+0000 7f662142d640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:50.169 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.166+0000 7f662142d640 10 monclient: dump: 2024-03-25T18:24:50.169 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:24:50.169 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:50.169 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:50.169 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:50.170 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:50.170 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:24:50.170 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:50.170 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:50.170 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:50.170 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:24:50.170 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.167+0000 7f6624693640 5 monclient: authenticate success, global_id 4174 2024-03-25T18:24:50.170 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.167+0000 7f6624693640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:50.173 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.169+0000 7f662142d640 1 -- 172.21.15.155:0/2536370689 <== mon.1 v1:172.21.15.167:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6614001ca0 con 0x7f661c1430c0 2024-03-25T18:24:50.173 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.169+0000 7f662142d640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:50.173 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.169+0000 7f6624693640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:50.173 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.169+0000 7f6624693640 4 set_mon_vals no callback set 2024-03-25T18:24:50.173 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.169+0000 7f662142d640 1 -- 172.21.15.155:0/2536370689 <== mon.1 v1:172.21.15.167:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f6614002030 con 0x7f661c1430c0 2024-03-25T18:24:50.173 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.169+0000 7f662142d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:50.173 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.169+0000 7f662142d640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:50.173 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.169+0000 7f662142d640 10 monclient: dump: 2024-03-25T18:24:50.173 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:24:50.173 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:50.173 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:50.173 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:50.174 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:50.174 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:24:50.174 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:50.174 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:50.174 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:50.174 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:24:50.174 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.170+0000 7f6624693640 10 monclient: shutdown 2024-03-25T18:24:50.174 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.170+0000 7f6624693640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:50.174 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.170+0000 7f6624693640 1 -- 172.21.15.155:0/2536370689 >> v1:172.21.15.167:6789/0 conn(0x7f661c1430c0 legacy=0x7f661c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:50.174 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.171+0000 7f6624693640 1 -- 172.21.15.155:0/2536370689 shutdown_connections 2024-03-25T18:24:50.174 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.171+0000 7f6624693640 1 -- 172.21.15.155:0/2536370689 >> 172.21.15.155:0/2536370689 conn(0x7f661c0b7ea0 msgr2=0x7f661c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:50.174 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.171+0000 7f6624693640 1 -- 172.21.15.155:0/2536370689 shutdown_connections 2024-03-25T18:24:50.174 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.171+0000 7f6624693640 1 -- 172.21.15.155:0/2536370689 wait complete. 2024-03-25T18:24:50.176 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.172+0000 7f6624693640 10 monclient: build_initial_monmap 2024-03-25T18:24:50.177 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.172+0000 7f6624693640 10 monclient: monmap: 2024-03-25T18:24:50.177 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:24:50.177 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:50.177 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:24:50.177 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:24:50.177 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:50.177 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:24:50.177 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:24:50.177 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:24:50.177 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:24:50.177 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:24:50.177 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.173+0000 7f6624693640 1 librados: starting msgr at 2024-03-25T18:24:50.177 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.173+0000 7f6624693640 1 librados: starting objecter 2024-03-25T18:24:50.177 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.173+0000 7f6624693640 1 Processor -- start 2024-03-25T18:24:50.177 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.173+0000 7f6624693640 1 -- start start 2024-03-25T18:24:50.178 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.173+0000 7f6624693640 1 librados: setting wanted keys 2024-03-25T18:24:50.178 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.173+0000 7f6624693640 1 librados: calling monclient init 2024-03-25T18:24:50.178 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.173+0000 7f6624693640 10 monclient: init 2024-03-25T18:24:50.178 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.174+0000 7f6624693640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:50.178 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.174+0000 7f6624693640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:24:50.178 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.174+0000 7f6624693640 10 monclient(hunting): picked mon.noname-a con 0x7f661c0a98f0 addr v1:172.21.15.155:6789/0 2024-03-25T18:24:50.178 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.174+0000 7f6624693640 10 monclient(hunting): picked mon.noname-c con 0x7f661c13f710 addr v1:172.21.15.155:6790/0 2024-03-25T18:24:50.178 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.174+0000 7f6624693640 10 monclient(hunting): picked mon.noname-b con 0x7f661c1430c0 addr v1:172.21.15.167:6789/0 2024-03-25T18:24:50.178 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.174+0000 7f6624693640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f661c06ff00 con 0x7f661c0a98f0 2024-03-25T18:24:50.178 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.174+0000 7f6624693640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f661c0701e0 con 0x7f661c1430c0 2024-03-25T18:24:50.178 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.174+0000 7f6624693640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f661c14d040 con 0x7f661c13f710 2024-03-25T18:24:50.178 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.174+0000 7f6624693640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:50.178 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.174+0000 7f6624693640 10 monclient(hunting): authenticate will time out at 7536.484211s 2024-03-25T18:24:50.179 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.178+0000 7f6622c30640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f661c1430c0 0x7f661c1e64e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:49048/0 (socket says 172.21.15.155:49048) 2024-03-25T18:24:50.179 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.178+0000 7f6622c30640 1 -- 172.21.15.155:0/2339487818 learned_addr learned my addr 172.21.15.155:0/2339487818 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:24:50.179 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.178+0000 7f6612ffd640 10 client.?.objecter ms_handle_connect 0x7f661c1430c0 2024-03-25T18:24:50.179 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.178+0000 7f6612ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:50.180 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.180+0000 7f6612ffd640 10 client.?.objecter ms_handle_connect 0x7f661c0a98f0 2024-03-25T18:24:50.180 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.180+0000 7f6612ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:50.183 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.182+0000 7f6612ffd640 1 -- 172.21.15.155:0/2339487818 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4081719263 0 0) 0x7f661c0701e0 con 0x7f661c1430c0 2024-03-25T18:24:50.183 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.182+0000 7f6612ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:50.183 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.182+0000 7f6612ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:50.183 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.182+0000 7f6612ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:50.183 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.182+0000 7f6612ffd640 10 monclient(hunting): my global_id is 4177 2024-03-25T18:24:50.183 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.182+0000 7f6612ffd640 1 -- 172.21.15.155:0/2339487818 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6600004eb0 con 0x7f661c1430c0 2024-03-25T18:24:50.185 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.184+0000 7f6612ffd640 10 client.?.objecter ms_handle_connect 0x7f661c13f710 2024-03-25T18:24:50.185 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.184+0000 7f6612ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:50.186 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.185+0000 7f6612ffd640 1 -- 172.21.15.155:0/2339487818 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2787644897 0 0) 0x7f661c06ff00 con 0x7f661c0a98f0 2024-03-25T18:24:50.186 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.185+0000 7f6612ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:50.186 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.185+0000 7f6612ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:50.187 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.185+0000 7f6612ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:50.187 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.185+0000 7f6612ffd640 10 monclient(hunting): my global_id is 4176 2024-03-25T18:24:50.187 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.185+0000 7f6612ffd640 1 -- 172.21.15.155:0/2339487818 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f661c0701e0 con 0x7f661c0a98f0 2024-03-25T18:24:50.191 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.190+0000 7f6612ffd640 1 -- 172.21.15.155:0/2339487818 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 348106791 0 0) 0x7f6600004eb0 con 0x7f661c1430c0 2024-03-25T18:24:50.191 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.190+0000 7f6612ffd640 1 -- 172.21.15.155:0/2339487818 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f661c06ff00 con 0x7f661c1430c0 2024-03-25T18:24:50.191 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.190+0000 7f6612ffd640 1 -- 172.21.15.155:0/2339487818 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f6614002030 con 0x7f661c1430c0 2024-03-25T18:24:50.191 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.190+0000 7f6612ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:50.191 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.190+0000 7f6612ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:50.191 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.190+0000 7f6612ffd640 10 monclient(hunting): dump: 2024-03-25T18:24:50.191 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:24:50.191 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:50.191 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:50.191 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:50.191 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:50.191 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:24:50.191 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:50.191 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:50.192 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:50.192 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:24:50.192 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.190+0000 7f6612ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:50.195 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.193+0000 7f6612ffd640 1 -- 172.21.15.155:0/2339487818 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2589862871 0 0) 0x7f661c0701e0 con 0x7f661c0a98f0 2024-03-25T18:24:50.195 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.193+0000 7f6612ffd640 1 -- 172.21.15.155:0/2339487818 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6600004eb0 con 0x7f661c0a98f0 2024-03-25T18:24:50.195 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.193+0000 7f6612ffd640 1 -- 172.21.15.155:0/2339487818 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f66180018b0 con 0x7f661c0a98f0 2024-03-25T18:24:50.195 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.193+0000 7f6612ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:50.195 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.193+0000 7f6612ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:50.195 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.193+0000 7f6612ffd640 10 monclient(hunting): dump: 2024-03-25T18:24:50.195 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:24:50.195 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:50.195 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:50.195 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:50.195 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:50.195 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:24:50.195 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:50.196 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:50.196 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:50.196 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:24:50.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.195+0000 7f6612ffd640 1 -- 172.21.15.155:0/2339487818 <== mon.1 v1:172.21.15.167:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 284647926 0 0) 0x7f661c06ff00 con 0x7f661c1430c0 2024-03-25T18:24:50.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.195+0000 7f6612ffd640 1 -- 172.21.15.155:0/2339487818 >> v1:172.21.15.155:6790/0 conn(0x7f661c13f710 legacy=0x7f661c0708e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:50.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.195+0000 7f6612ffd640 1 -- 172.21.15.155:0/2339487818 >> v1:172.21.15.155:6789/0 conn(0x7f661c0a98f0 legacy=0x7f661c06eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:50.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.195+0000 7f6612ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:24:50.197 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.195+0000 7f6612ffd640 1 monclient: found mon.b 2024-03-25T18:24:50.197 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.195+0000 7f6612ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:50.197 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.195+0000 7f6612ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:24:50.197 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.195+0000 7f6612ffd640 1 -- 172.21.15.155:0/2339487818 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f661c14d540 con 0x7f661c1430c0 2024-03-25T18:24:50.197 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.195+0000 7f6612ffd640 10 monclient: _finish_auth 0 2024-03-25T18:24:50.197 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.195+0000 7f6612ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:24:50.197 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.195+0000 7f6612ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:50.197 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.195+0000 7f6624693640 5 monclient: authenticate success, global_id 4177 2024-03-25T18:24:50.197 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.195+0000 7f6624693640 10 monclient: _renew_subs 2024-03-25T18:24:50.197 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.195+0000 7f6624693640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:24:50.197 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.195+0000 7f6624693640 1 -- 172.21.15.155:0/2339487818 --> v1:172.21.15.167:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f661c06f2a0 con 0x7f661c1430c0 2024-03-25T18:24:50.197 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.196+0000 7f6624693640 10 client.4177.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:50.197 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.196+0000 7f6624693640 10 monclient: _renew_subs 2024-03-25T18:24:50.197 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.196+0000 7f6624693640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:24:50.198 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.196+0000 7f6624693640 1 -- 172.21.15.155:0/2339487818 --> v1:172.21.15.167:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f661c0711d0 con 0x7f661c1430c0 2024-03-25T18:24:50.198 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.196+0000 7f6624693640 1 librados: init done 2024-03-25T18:24:50.198 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.197+0000 7f6624693640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:50.198 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.197+0000 7f6624693640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:50.198 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.197+0000 7f6624693640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:24:50.198 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.197+0000 7f6624693640 1 -- 172.21.15.155:0/2339487818 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f661c0714d0 con 0x7f661c1430c0 2024-03-25T18:24:50.216 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.215+0000 7f6612ffd640 1 -- 172.21.15.155:0/2339487818 <== mon.1 v1:172.21.15.167:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6614001500 con 0x7f661c1430c0 2024-03-25T18:24:50.217 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.215+0000 7f6612ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:50.217 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.215+0000 7f6612ffd640 1 -- 172.21.15.155:0/2339487818 <== mon.1 v1:172.21.15.167:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f6614004fb0 con 0x7f661c1430c0 2024-03-25T18:24:50.217 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.215+0000 7f6612ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:50.217 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.215+0000 7f6612ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:50.217 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.215+0000 7f6612ffd640 10 monclient: dump: 2024-03-25T18:24:50.217 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:24:50.217 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:50.217 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:50.217 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:50.217 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:50.217 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:24:50.217 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:50.217 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:50.217 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:50.217 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:24:50.218 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.215+0000 7f6612ffd640 1 -- 172.21.15.155:0/2339487818 <== mon.1 v1:172.21.15.167:6789/0 7 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 3719313413 0 0) 0x7f6614010090 con 0x7f661c1430c0 2024-03-25T18:24:50.218 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.216+0000 7f6620c2c640 4 set_mon_vals no callback set 2024-03-25T18:24:50.219 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.217+0000 7f6612ffd640 1 -- 172.21.15.155:0/2339487818 <== mon.1 v1:172.21.15.167:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7f6614015cb0 con 0x7f661c1430c0 2024-03-25T18:24:50.219 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.217+0000 7f6612ffd640 10 client.4177.objecter ms_dispatch 0x7f661c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:24:50.219 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.217+0000 7f6612ffd640 3 client.4177.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:24:50.219 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.217+0000 7f6612ffd640 3 client.4177.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:24:50.219 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.217+0000 7f6612ffd640 20 client.4177.objecter dump_active .. 0 homeless 2024-03-25T18:24:50.393 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.392+0000 7f6612ffd640 1 -- 172.21.15.155:0/2339487818 <== mon.1 v1:172.21.15.167:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f6614015150 con 0x7f661c1430c0 2024-03-25T18:24:50.393 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.392+0000 7f6612ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:50.393 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.392+0000 7f6612ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:50.518 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.517+0000 7f6624693640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:50.518 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.517+0000 7f6624693640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:50.518 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.517+0000 7f6624693640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:24:50.518 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.517+0000 7f6624693640 1 -- 172.21.15.155:0/2339487818 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f661c14d1d0 con 0x7f661c1430c0 2024-03-25T18:24:50.884 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.882+0000 7f6612ffd640 1 -- 172.21.15.155:0/2339487818 <== mon.1 v1:172.21.15.167:6789/0 10 ==== mgrmap(e 3) v1 ==== 89591+0+0 (unknown 4009683039 0 0) 0x7f6614011c10 con 0x7f661c1430c0 2024-03-25T18:24:50.885 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.883+0000 7f662242f640 10 monclient: get_auth_request con 0x7f660006e240 auth_method 0 2024-03-25T18:24:50.885 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.884+0000 7f6612ffd640 10 client.4177.objecter ms_handle_connect 0x7f660006e240 2024-03-25T18:24:50.929 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.929+0000 7f6612ffd640 1 -- 172.21.15.155:0/2339487818 <== mon.1 v1:172.21.15.167:6789/0 11 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7f6614027cd0 con 0x7f661c1430c0 2024-03-25T18:24:50.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.929+0000 7f6612ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:50.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.929+0000 7f6612ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:50.930 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:24:50.930 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:50.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.930+0000 7f6624693640 10 librados: watch_flush enter 2024-03-25T18:24:50.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.930+0000 7f6624693640 10 librados: watch_flush exit 2024-03-25T18:24:50.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.930+0000 7f6624693640 20 client.4177.objecter shutdown clearing up homeless session... 2024-03-25T18:24:50.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.930+0000 7f6624693640 10 client.4177.objecter successfully canceled tick 2024-03-25T18:24:50.932 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.930+0000 7f6624693640 1 -- 172.21.15.155:0/2339487818 >> v1:172.21.15.167:6800/2919070503 conn(0x7f660006e240 legacy=0x7f6600070630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:50.932 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.930+0000 7f6624693640 10 monclient: shutdown 2024-03-25T18:24:50.932 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.930+0000 7f6624693640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:50.932 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.930+0000 7f6624693640 1 -- 172.21.15.155:0/2339487818 >> v1:172.21.15.167:6789/0 conn(0x7f661c1430c0 legacy=0x7f661c1e64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:50.932 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.930+0000 7f6624693640 1 -- 172.21.15.155:0/2339487818 shutdown_connections 2024-03-25T18:24:50.932 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.930+0000 7f6624693640 1 -- 172.21.15.155:0/2339487818 >> 172.21.15.155:0/2339487818 conn(0x7f661c0b7ea0 msgr2=0x7f661c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:50.932 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.931+0000 7f6624693640 1 -- 172.21.15.155:0/2339487818 shutdown_connections 2024-03-25T18:24:50.932 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.931+0000 7f6624693640 1 -- 172.21.15.155:0/2339487818 wait complete. 2024-03-25T18:24:50.932 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:50.931+0000 7f6624693640 1 librados: shutdown 2024-03-25T18:24:50.952 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:50.985 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-03-25T18:24:50.984+0000 7f17c13ae640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-25T18:24:51.013 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-03-25T18:24:51.013+0000 7f17c13ae640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-25T18:24:55.394 INFO:tasks.ceph.osd.6.smithi167.stderr:==00:00:00:06.764 35185== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:55.455 INFO:tasks.ceph.osd.6.smithi167.stderr:==00:00:00:06.825 35185== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:55.512 INFO:tasks.ceph.osd.7.smithi167.stderr:==00:00:00:06.878 35186== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:55.535 INFO:tasks.ceph.osd.5.smithi167.stderr:==00:00:00:06.909 35184== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:55.577 INFO:tasks.ceph.osd.7.smithi167.stderr:==00:00:00:06.944 35186== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:55.596 INFO:tasks.ceph.osd.5.smithi167.stderr:==00:00:00:06.969 35184== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:55.914 INFO:tasks.ceph.osd.1.smithi155.stderr:==00:00:00:07.279 35525== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:55.951 INFO:tasks.ceph.osd.4.smithi167.stderr:==00:00:00:07.306 35187== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:55.975 INFO:tasks.ceph.osd.1.smithi155.stderr:==00:00:00:07.340 35525== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:56.011 INFO:tasks.ceph.osd.4.smithi167.stderr:==00:00:00:07.365 35187== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:56.026 INFO:tasks.ceph.osd.0.smithi155.stderr:==00:00:00:07.398 35524== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:56.031 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-03-25T18:24:56.025+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:24:56.078 INFO:tasks.ceph.osd.2.smithi155.stderr:==00:00:00:07.460 35523== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:56.087 INFO:tasks.ceph.osd.0.smithi155.stderr:==00:00:00:07.459 35524== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:56.128 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-03-25T18:24:56.121+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:24:56.135 INFO:tasks.ceph.osd.3.smithi155.stderr:==00:00:00:07.494 35526== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:56.138 INFO:tasks.ceph.osd.2.smithi155.stderr:==00:00:00:07.520 35523== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:56.147 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-03-25T18:24:56.140+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:24:56.195 INFO:tasks.ceph.osd.3.smithi155.stderr:==00:00:00:07.554 35526== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:56.512 INFO:tasks.ceph.osd.6.smithi167.stderr:==00:00:00:07.881 35185== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:56.528 INFO:tasks.ceph.osd.1.smithi155.stderr:2024-03-25T18:24:56.522+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:24:56.589 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-03-25T18:24:56.583+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:24:56.590 INFO:tasks.ceph.osd.7.smithi167.stderr:==00:00:00:07.956 35186== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:56.617 INFO:tasks.ceph.osd.5.smithi167.stderr:==00:00:00:07.991 35184== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:56.655 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-03-25T18:24:56.647+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:24:56.687 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-03-25T18:24:56.677+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:24:56.766 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-03-25T18:24:56.759+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:24:56.897 INFO:tasks.ceph.osd.6.smithi167.stderr:==00:00:00:08.266 35185== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:56.990 INFO:tasks.ceph.osd.5.smithi167.stderr:==00:00:00:08.363 35184== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:56.995 INFO:tasks.ceph.osd.7.smithi167.stderr:==00:00:00:08.362 35186== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:57.015 INFO:tasks.ceph.osd.1.smithi155.stderr:==00:00:00:08.381 35525== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:57.101 INFO:tasks.ceph.osd.4.smithi167.stderr:==00:00:00:08.456 35187== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:57.166 INFO:tasks.ceph.osd.2.smithi155.stderr:==00:00:00:08.547 35523== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:57.166 INFO:tasks.ceph.osd.0.smithi155.stderr:==00:00:00:08.538 35524== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:57.241 INFO:tasks.ceph.osd.3.smithi155.stderr:==00:00:00:08.600 35526== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:57.256 INFO:tasks.ceph.osd.6.smithi167.stderr:==00:00:00:08.626 35185== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:57.322 INFO:tasks.ceph.osd.7.smithi167.stderr:==00:00:00:08.689 35186== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:57.353 INFO:tasks.ceph.osd.5.smithi167.stderr:==00:00:00:08.727 35184== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:57.377 INFO:tasks.ceph.osd.1.smithi155.stderr:==00:00:00:08.742 35525== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:57.467 INFO:tasks.ceph.osd.4.smithi167.stderr:==00:00:00:08.821 35187== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:57.534 INFO:tasks.ceph.osd.2.smithi155.stderr:==00:00:00:08.915 35523== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:57.538 INFO:tasks.ceph.osd.0.smithi155.stderr:==00:00:00:08.910 35524== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:57.544 INFO:tasks.ceph.osd.6.smithi167.stderr:==00:00:00:08.914 35185== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:57.591 INFO:tasks.ceph.osd.7.smithi167.stderr:==00:00:00:08.958 35186== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:57.608 INFO:tasks.ceph.osd.3.smithi155.stderr:==00:00:00:08.967 35526== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:57.641 INFO:tasks.ceph.osd.5.smithi167.stderr:==00:00:00:09.014 35184== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:57.727 INFO:tasks.ceph.osd.1.smithi155.stderr:==00:00:00:09.093 35525== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:57.758 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:57.777 INFO:tasks.ceph.osd.4.smithi167.stderr:==00:00:00:09.131 35187== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:57.809 INFO:tasks.ceph.osd.6.smithi167.stderr:==00:00:00:09.178 35185== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:57.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.845+0000 7f1a9ec57640 10 monclient: get_monmap_and_config 2024-03-25T18:24:57.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.845+0000 7f1a9ec57640 10 monclient: build_initial_monmap 2024-03-25T18:24:57.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.845+0000 7f1a9ec57640 10 monclient: monmap: 2024-03-25T18:24:57.847 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:24:57.847 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:57.847 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:24:57.846848+0000 2024-03-25T18:24:57.847 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:24:57.846848+0000 2024-03-25T18:24:57.847 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:57.847 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:24:57.847 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:24:57.847 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:24:57.847 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:24:57.847 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:24:57.848 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.846+0000 7f1a9ec57640 1 Processor -- start 2024-03-25T18:24:57.849 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.846+0000 7f1a9ec57640 1 -- start start 2024-03-25T18:24:57.849 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.846+0000 7f1a9ec57640 10 monclient: init 2024-03-25T18:24:57.849 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.847+0000 7f1a9ec57640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:57.849 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.847+0000 7f1a9ec57640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:24:57.849 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.847+0000 7f1a9ec57640 10 monclient(hunting): picked mon.noname-b con 0x7f1a980704a0 addr v1:172.21.15.155:6790/0 2024-03-25T18:24:57.849 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.847+0000 7f1a9ec57640 10 monclient(hunting): picked mon.noname-a con 0x7f1a98070880 addr v1:172.21.15.155:6789/0 2024-03-25T18:24:57.849 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.847+0000 7f1a9ec57640 10 monclient(hunting): picked mon.noname-c con 0x7f1a98149a40 addr v1:172.21.15.167:6789/0 2024-03-25T18:24:57.849 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.847+0000 7f1a9ec57640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a98143870 con 0x7f1a98070880 2024-03-25T18:24:57.849 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.847+0000 7f1a9ec57640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a98072480 con 0x7f1a98149a40 2024-03-25T18:24:57.849 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.847+0000 7f1a9ec57640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a98072760 con 0x7f1a980704a0 2024-03-25T18:24:57.849 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.847+0000 7f1a9ec57640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:57.849 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.847+0000 7f1a9ec57640 10 monclient(hunting): authenticate will time out at 7544.156959s 2024-03-25T18:24:57.849 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.847+0000 7f1a9c9f3640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f1a980704a0 0x7f1a98143d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:60224/0 (socket says 172.21.15.155:60224) 2024-03-25T18:24:57.850 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.847+0000 7f1a9d1f4640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f1a98149a40 0x7f1a9814be30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:45672/0 (socket says 172.21.15.155:45672) 2024-03-25T18:24:57.850 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.847+0000 7f1a9c9f3640 1 -- 172.21.15.155:0/2945068517 learned_addr learned my addr 172.21.15.155:0/2945068517 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:24:57.851 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.850+0000 7f1a977fe640 1 -- 172.21.15.155:0/2945068517 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1821140759 0 0) 0x7f1a98072480 con 0x7f1a98149a40 2024-03-25T18:24:57.851 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.850+0000 7f1a977fe640 10 monclient(hunting): _negotiate 2024-03-25T18:24:57.851 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.850+0000 7f1a977fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:57.851 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.850+0000 7f1a977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:57.851 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.850+0000 7f1a977fe640 10 monclient(hunting): my global_id is 4204 2024-03-25T18:24:57.851 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.850+0000 7f1a977fe640 1 -- 172.21.15.155:0/2945068517 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a88003520 con 0x7f1a98149a40 2024-03-25T18:24:57.851 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.850+0000 7f1a977fe640 1 -- 172.21.15.155:0/2945068517 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2582907446 0 0) 0x7f1a98072760 con 0x7f1a980704a0 2024-03-25T18:24:57.851 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.850+0000 7f1a977fe640 10 monclient(hunting): _negotiate 2024-03-25T18:24:57.851 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.850+0000 7f1a977fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:57.851 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.850+0000 7f1a977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:57.851 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.850+0000 7f1a977fe640 10 monclient(hunting): my global_id is 4205 2024-03-25T18:24:57.851 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.850+0000 7f1a977fe640 1 -- 172.21.15.155:0/2945068517 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a98072480 con 0x7f1a980704a0 2024-03-25T18:24:57.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.850+0000 7f1a977fe640 1 -- 172.21.15.155:0/2945068517 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1549839162 0 0) 0x7f1a98143870 con 0x7f1a98070880 2024-03-25T18:24:57.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.850+0000 7f1a977fe640 10 monclient(hunting): _negotiate 2024-03-25T18:24:57.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.850+0000 7f1a977fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:57.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.850+0000 7f1a977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:57.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.850+0000 7f1a977fe640 10 monclient(hunting): my global_id is 4203 2024-03-25T18:24:57.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.850+0000 7f1a977fe640 1 -- 172.21.15.155:0/2945068517 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a98072760 con 0x7f1a98070880 2024-03-25T18:24:57.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.853+0000 7f1a977fe640 1 -- 172.21.15.155:0/2945068517 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3538173036 0 0) 0x7f1a88003520 con 0x7f1a98149a40 2024-03-25T18:24:57.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.853+0000 7f1a977fe640 1 -- 172.21.15.155:0/2945068517 >> v1:172.21.15.155:6790/0 conn(0x7f1a980704a0 legacy=0x7f1a98143d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:57.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.853+0000 7f1a977fe640 1 -- 172.21.15.155:0/2945068517 >> v1:172.21.15.155:6789/0 conn(0x7f1a98070880 legacy=0x7f1a9813d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:57.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.853+0000 7f1a977fe640 10 monclient: _finish_hunting 0 2024-03-25T18:24:57.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.853+0000 7f1a977fe640 1 monclient: found mon.noname-c 2024-03-25T18:24:57.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.853+0000 7f1a977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:57.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.853+0000 7f1a977fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.167:6789/0 2024-03-25T18:24:57.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.853+0000 7f1a977fe640 1 -- 172.21.15.155:0/2945068517 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a98072a40 con 0x7f1a98149a40 2024-03-25T18:24:57.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.853+0000 7f1a977fe640 10 monclient: _finish_auth 0 2024-03-25T18:24:57.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.853+0000 7f1a977fe640 10 monclient: _check_auth_tickets 2024-03-25T18:24:57.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.853+0000 7f1a977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:57.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.853+0000 7f1a977fe640 1 -- 172.21.15.155:0/2945068517 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f1a90001a10 con 0x7f1a98149a40 2024-03-25T18:24:57.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.853+0000 7f1a9ec57640 5 monclient: authenticate success, global_id 4204 2024-03-25T18:24:57.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.853+0000 7f1a9ec57640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:57.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.853+0000 7f1a977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:57.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.853+0000 7f1a977fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:57.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.853+0000 7f1a977fe640 10 monclient: dump: 2024-03-25T18:24:57.855 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:24:57.855 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:57.856 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:57.856 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:57.856 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:57.856 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:24:57.856 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:57.856 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:57.856 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:57.856 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:24:57.856 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.853+0000 7f1a9ec57640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:57.857 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.854+0000 7f1a977fe640 1 -- 172.21.15.155:0/2945068517 <== mon.1 v1:172.21.15.167:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1a90001b70 con 0x7f1a98149a40 2024-03-25T18:24:57.857 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.854+0000 7f1a977fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:57.857 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.854+0000 7f1a9ec57640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:57.857 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.854+0000 7f1a9ec57640 4 set_mon_vals no callback set 2024-03-25T18:24:57.857 INFO:tasks.ceph.osd.2.smithi155.stderr:==00:00:00:09.237 35523== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:57.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.855+0000 7f1a977fe640 1 -- 172.21.15.155:0/2945068517 <== mon.1 v1:172.21.15.167:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f1a90001f00 con 0x7f1a98149a40 2024-03-25T18:24:57.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.855+0000 7f1a977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:57.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.855+0000 7f1a977fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:57.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.855+0000 7f1a977fe640 10 monclient: dump: 2024-03-25T18:24:57.860 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:24:57.860 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:57.861 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:57.861 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:57.861 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:57.861 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:24:57.861 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:57.861 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:57.861 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:57.861 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:24:57.861 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.855+0000 7f1a9ec57640 10 monclient: shutdown 2024-03-25T18:24:57.861 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.855+0000 7f1a9ec57640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:57.861 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.855+0000 7f1a9ec57640 1 -- 172.21.15.155:0/2945068517 >> v1:172.21.15.167:6789/0 conn(0x7f1a98149a40 legacy=0x7f1a9814be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:57.861 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.855+0000 7f1a9ec57640 1 -- 172.21.15.155:0/2945068517 shutdown_connections 2024-03-25T18:24:57.861 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.855+0000 7f1a9ec57640 1 -- 172.21.15.155:0/2945068517 >> 172.21.15.155:0/2945068517 conn(0x7f1a980b5c00 msgr2=0x7f1a980b5fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:57.861 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.855+0000 7f1a9ec57640 1 -- 172.21.15.155:0/2945068517 shutdown_connections 2024-03-25T18:24:57.862 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.855+0000 7f1a9ec57640 1 -- 172.21.15.155:0/2945068517 wait complete. 2024-03-25T18:24:57.862 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.856+0000 7f1a9ec57640 10 monclient: build_initial_monmap 2024-03-25T18:24:57.862 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.856+0000 7f1a9ec57640 10 monclient: monmap: 2024-03-25T18:24:57.862 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:24:57.862 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:57.862 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:24:57.862 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:24:57.862 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:57.862 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:24:57.862 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:24:57.862 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:24:57.862 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:24:57.862 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:24:57.862 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.856+0000 7f1a9ec57640 1 librados: starting msgr at 2024-03-25T18:24:57.862 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.856+0000 7f1a9ec57640 1 librados: starting objecter 2024-03-25T18:24:57.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.856+0000 7f1a9ec57640 1 Processor -- start 2024-03-25T18:24:57.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.856+0000 7f1a9ec57640 1 -- start start 2024-03-25T18:24:57.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.856+0000 7f1a9ec57640 1 librados: setting wanted keys 2024-03-25T18:24:57.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.856+0000 7f1a9ec57640 1 librados: calling monclient init 2024-03-25T18:24:57.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.856+0000 7f1a9ec57640 10 monclient: init 2024-03-25T18:24:57.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.857+0000 7f1a9ec57640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:57.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.857+0000 7f1a9ec57640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:24:57.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.857+0000 7f1a9ec57640 10 monclient(hunting): picked mon.noname-b con 0x7f1a980704a0 addr v1:172.21.15.167:6789/0 2024-03-25T18:24:57.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.857+0000 7f1a9ec57640 10 monclient(hunting): picked mon.noname-a con 0x7f1a98070880 addr v1:172.21.15.155:6789/0 2024-03-25T18:24:57.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.857+0000 7f1a9ec57640 10 monclient(hunting): picked mon.noname-c con 0x7f1a98149a40 addr v1:172.21.15.155:6790/0 2024-03-25T18:24:57.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.857+0000 7f1a9ec57640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a98144830 con 0x7f1a98070880 2024-03-25T18:24:57.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.857+0000 7f1a9ec57640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a98145ec0 con 0x7f1a980704a0 2024-03-25T18:24:57.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.857+0000 7f1a9ec57640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a981d9640 con 0x7f1a98149a40 2024-03-25T18:24:57.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.857+0000 7f1a9ec57640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:57.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.857+0000 7f1a9ec57640 10 monclient(hunting): authenticate will time out at 7544.167542s 2024-03-25T18:24:57.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.858+0000 7f1a9d1f4640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f1a98149a40 0x7f1a981dfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:60230/0 (socket says 172.21.15.155:60230) 2024-03-25T18:24:57.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.858+0000 7f1a9d1f4640 1 -- 172.21.15.155:0/2877958506 learned_addr learned my addr 172.21.15.155:0/2877958506 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:24:57.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.858+0000 7f1a957fa640 10 client.?.objecter ms_handle_connect 0x7f1a98149a40 2024-03-25T18:24:57.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.858+0000 7f1a957fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:57.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.858+0000 7f1a957fa640 10 client.?.objecter ms_handle_connect 0x7f1a980704a0 2024-03-25T18:24:57.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.858+0000 7f1a957fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:57.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.858+0000 7f1a957fa640 10 client.?.objecter ms_handle_connect 0x7f1a98070880 2024-03-25T18:24:57.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.858+0000 7f1a957fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:57.865 INFO:tasks.ceph.osd.7.smithi167.stderr:==00:00:00:09.230 35186== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:57.865 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.860+0000 7f1a957fa640 1 -- 172.21.15.155:0/2877958506 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3039443514 0 0) 0x7f1a98145ec0 con 0x7f1a980704a0 2024-03-25T18:24:57.865 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.860+0000 7f1a957fa640 10 monclient(hunting): _negotiate 2024-03-25T18:24:57.865 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.860+0000 7f1a957fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:57.866 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.860+0000 7f1a957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:57.866 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.860+0000 7f1a957fa640 10 monclient(hunting): my global_id is 4207 2024-03-25T18:24:57.866 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.860+0000 7f1a957fa640 1 -- 172.21.15.155:0/2877958506 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a7c004eb0 con 0x7f1a980704a0 2024-03-25T18:24:57.866 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.860+0000 7f1a957fa640 1 -- 172.21.15.155:0/2877958506 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3316860607 0 0) 0x7f1a98144830 con 0x7f1a98070880 2024-03-25T18:24:57.866 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.860+0000 7f1a957fa640 10 monclient(hunting): _negotiate 2024-03-25T18:24:57.866 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.860+0000 7f1a957fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:57.866 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.860+0000 7f1a957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:57.866 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.860+0000 7f1a957fa640 10 monclient(hunting): my global_id is 4206 2024-03-25T18:24:57.866 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.860+0000 7f1a957fa640 1 -- 172.21.15.155:0/2877958506 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a98145ec0 con 0x7f1a98070880 2024-03-25T18:24:57.866 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.861+0000 7f1a957fa640 1 -- 172.21.15.155:0/2877958506 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 756626114 0 0) 0x7f1a981d9640 con 0x7f1a98149a40 2024-03-25T18:24:57.866 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.861+0000 7f1a957fa640 10 monclient(hunting): _negotiate 2024-03-25T18:24:57.866 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.861+0000 7f1a957fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:57.866 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.861+0000 7f1a957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:57.866 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.861+0000 7f1a957fa640 10 monclient(hunting): my global_id is 4208 2024-03-25T18:24:57.866 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.861+0000 7f1a957fa640 1 -- 172.21.15.155:0/2877958506 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a98144830 con 0x7f1a98149a40 2024-03-25T18:24:57.868 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.865+0000 7f1a957fa640 1 -- 172.21.15.155:0/2877958506 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4117414852 0 0) 0x7f1a7c004eb0 con 0x7f1a980704a0 2024-03-25T18:24:57.868 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.865+0000 7f1a957fa640 1 -- 172.21.15.155:0/2877958506 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1a981d9640 con 0x7f1a980704a0 2024-03-25T18:24:57.868 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.865+0000 7f1a957fa640 1 -- 172.21.15.155:0/2877958506 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 476148130 0 0) 0x7f1a98144830 con 0x7f1a98149a40 2024-03-25T18:24:57.868 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.865+0000 7f1a957fa640 1 -- 172.21.15.155:0/2877958506 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1a7c004eb0 con 0x7f1a98149a40 2024-03-25T18:24:57.868 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.865+0000 7f1a957fa640 1 -- 172.21.15.155:0/2877958506 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f1a90001e80 con 0x7f1a98149a40 2024-03-25T18:24:57.868 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.865+0000 7f1a957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:57.868 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.865+0000 7f1a957fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:57.868 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.865+0000 7f1a957fa640 10 monclient(hunting): dump: 2024-03-25T18:24:57.868 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:24:57.868 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:57.868 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:57.868 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:57.868 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:57.868 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:24:57.869 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:57.869 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:57.869 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:57.869 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:24:57.869 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.865+0000 7f1a957fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:57.869 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.866+0000 7f1a957fa640 1 -- 172.21.15.155:0/2877958506 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f1a8c001a10 con 0x7f1a980704a0 2024-03-25T18:24:57.869 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.866+0000 7f1a957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:57.869 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.866+0000 7f1a957fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:57.869 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.866+0000 7f1a957fa640 10 monclient(hunting): dump: 2024-03-25T18:24:57.869 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:24:57.869 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:57.869 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:57.869 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:57.869 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:57.869 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:24:57.870 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:57.870 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:57.870 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:57.870 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:24:57.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.866+0000 7f1a957fa640 1 -- 172.21.15.155:0/2877958506 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 556023368 0 0) 0x7f1a98145ec0 con 0x7f1a98070880 2024-03-25T18:24:57.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.866+0000 7f1a957fa640 1 -- 172.21.15.155:0/2877958506 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1a98144830 con 0x7f1a98070880 2024-03-25T18:24:57.871 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.869+0000 7f1a957fa640 1 -- 172.21.15.155:0/2877958506 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2592653342 0 0) 0x7f1a7c004eb0 con 0x7f1a98149a40 2024-03-25T18:24:57.871 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.869+0000 7f1a957fa640 1 -- 172.21.15.155:0/2877958506 >> v1:172.21.15.167:6789/0 conn(0x7f1a980704a0 legacy=0x7f1a981457b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:57.871 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.869+0000 7f1a957fa640 1 -- 172.21.15.155:0/2877958506 >> v1:172.21.15.155:6789/0 conn(0x7f1a98070880 legacy=0x7f1a981d8f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:57.871 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.869+0000 7f1a957fa640 10 monclient: _finish_hunting 0 2024-03-25T18:24:57.871 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.869+0000 7f1a957fa640 1 monclient: found mon.c 2024-03-25T18:24:57.871 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.869+0000 7f1a957fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:57.871 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.869+0000 7f1a957fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:24:57.872 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.869+0000 7f1a957fa640 1 -- 172.21.15.155:0/2877958506 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a981d97d0 con 0x7f1a98149a40 2024-03-25T18:24:57.872 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.869+0000 7f1a957fa640 10 monclient: _finish_auth 0 2024-03-25T18:24:57.872 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.869+0000 7f1a957fa640 10 monclient: _check_auth_tickets 2024-03-25T18:24:57.872 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.869+0000 7f1a957fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:57.872 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.869+0000 7f1a9ec57640 5 monclient: authenticate success, global_id 4208 2024-03-25T18:24:57.872 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.869+0000 7f1a9ec57640 10 monclient: _renew_subs 2024-03-25T18:24:57.872 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.869+0000 7f1a9ec57640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:24:57.872 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.869+0000 7f1a9ec57640 1 -- 172.21.15.155:0/2877958506 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a9813eb00 con 0x7f1a98149a40 2024-03-25T18:24:57.872 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.869+0000 7f1a9ec57640 10 client.4208.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:57.872 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.869+0000 7f1a9ec57640 10 monclient: _renew_subs 2024-03-25T18:24:57.872 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.869+0000 7f1a9ec57640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:24:57.872 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.869+0000 7f1a9ec57640 1 -- 172.21.15.155:0/2877958506 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a98144cf0 con 0x7f1a98149a40 2024-03-25T18:24:57.872 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.869+0000 7f1a9ec57640 1 librados: init done 2024-03-25T18:24:57.872 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.870+0000 7f1a9ec57640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:57.873 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.870+0000 7f1a9ec57640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:57.873 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.870+0000 7f1a9ec57640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:24:57.873 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:57.870+0000 7f1a9ec57640 1 -- 172.21.15.155:0/2877958506 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a981d88b0 con 0x7f1a98149a40 2024-03-25T18:24:57.907 INFO:tasks.ceph.osd.0.smithi155.stderr:==00:00:00:09.279 35524== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:57.929 INFO:tasks.ceph.osd.5.smithi167.stderr:==00:00:00:09.302 35184== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:57.965 INFO:tasks.ceph.osd.3.smithi155.stderr:==00:00:00:09.324 35526== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:58.018 INFO:tasks.ceph.osd.1.smithi155.stderr:==00:00:00:09.383 35525== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:58.047 INFO:tasks.ceph.osd.4.smithi167.stderr:==00:00:00:09.402 35187== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:58.096 INFO:tasks.ceph.osd.6.smithi167.stderr:==00:00:00:09.465 35185== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:58.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:58.116+0000 7f1a957fa640 1 -- 172.21.15.155:0/2877958506 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1a90001fe0 con 0x7f1a98149a40 2024-03-25T18:24:58.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:58.116+0000 7f1a957fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:58.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:58.116+0000 7f1a957fa640 1 -- 172.21.15.155:0/2877958506 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f1a900015d0 con 0x7f1a98149a40 2024-03-25T18:24:58.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:58.116+0000 7f1a957fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:58.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:58.116+0000 7f1a957fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:58.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:58.116+0000 7f1a957fa640 10 monclient: dump: 2024-03-25T18:24:58.117 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:24:58.117 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:24:58.117 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:58.117 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:24:58.117 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:58.117 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:24:58.117 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:24:58.117 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:24:58.118 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:24:58.118 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:24:58.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:58.116+0000 7f1a96ffd640 4 set_mon_vals no callback set 2024-03-25T18:24:58.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:58.117+0000 7f1a957fa640 1 -- 172.21.15.155:0/2877958506 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7f1a9001ac80 con 0x7f1a98149a40 2024-03-25T18:24:58.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:58.117+0000 7f1a9c9f3640 10 monclient: get_auth_request con 0x7f1a7c06e220 auth_method 0 2024-03-25T18:24:58.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:58.118+0000 7f1a957fa640 10 client.4208.objecter ms_handle_connect 0x7f1a7c06e220 2024-03-25T18:24:58.121 INFO:tasks.ceph.osd.2.smithi155.stderr:==00:00:00:09.502 35523== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:58.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:58.121+0000 7f1a957fa640 1 -- 172.21.15.155:0/2877958506 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7f1a900548a0 con 0x7f1a98149a40 2024-03-25T18:24:58.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:58.121+0000 7f1a957fa640 10 client.4208.objecter ms_dispatch 0x7f1a98000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:24:58.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:58.121+0000 7f1a957fa640 3 client.4208.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:24:58.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:58.121+0000 7f1a957fa640 3 client.4208.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:24:58.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:58.121+0000 7f1a957fa640 20 client.4208.objecter dump_active .. 0 homeless 2024-03-25T18:24:58.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:58.121+0000 7f1a957fa640 1 -- 172.21.15.155:0/2877958506 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1a900503b0 con 0x7f1a98149a40 2024-03-25T18:24:58.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:58.121+0000 7f1a957fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:58.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:58.121+0000 7f1a957fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:58.135 INFO:tasks.ceph.osd.7.smithi167.stderr:==00:00:00:09.502 35186== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:58.201 INFO:tasks.ceph.osd.0.smithi155.stderr:==00:00:00:09.572 35524== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:58.220 INFO:tasks.ceph.osd.5.smithi167.stderr:==00:00:00:09.593 35184== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:58.258 INFO:tasks.ceph.osd.3.smithi155.stderr:==00:00:00:09.617 35526== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:58.305 INFO:tasks.ceph.osd.1.smithi155.stderr:==00:00:00:09.671 35525== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:58.315 INFO:tasks.ceph.osd.4.smithi167.stderr:==00:00:00:09.670 35187== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:58.325 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:58.324+0000 7f1a9ec57640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:58.325 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:58.324+0000 7f1a9ec57640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:58.325 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:58.324+0000 7f1a9ec57640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:24:58.325 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:58.324+0000 7f1a9ec57640 1 -- 172.21.15.155:0/2877958506 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1a9813ec90 con 0x7f1a98149a40 2024-03-25T18:24:58.368 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:58.368+0000 7f1a957fa640 1 -- 172.21.15.155:0/2877958506 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7f1a900522e0 con 0x7f1a98149a40 2024-03-25T18:24:58.369 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:58.368+0000 7f1a957fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:58.369 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:58.368+0000 7f1a957fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:58.369 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:24:58.369 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:58.371 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:58.370+0000 7f1a9ec57640 10 librados: watch_flush enter 2024-03-25T18:24:58.372 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:58.370+0000 7f1a9ec57640 10 librados: watch_flush exit 2024-03-25T18:24:58.372 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:58.370+0000 7f1a9ec57640 20 client.4208.objecter shutdown clearing up homeless session... 2024-03-25T18:24:58.372 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:58.370+0000 7f1a9ec57640 10 client.4208.objecter successfully canceled tick 2024-03-25T18:24:58.372 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:58.370+0000 7f1a9ec57640 1 -- 172.21.15.155:0/2877958506 >> v1:172.21.15.167:6800/2919070503 conn(0x7f1a7c06e220 legacy=0x7f1a7c0706e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:58.372 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:58.370+0000 7f1a9ec57640 10 monclient: shutdown 2024-03-25T18:24:58.372 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:58.370+0000 7f1a9ec57640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:58.372 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:58.370+0000 7f1a9ec57640 1 -- 172.21.15.155:0/2877958506 >> v1:172.21.15.155:6790/0 conn(0x7f1a98149a40 legacy=0x7f1a981dfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:58.372 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:58.370+0000 7f1a9ec57640 1 -- 172.21.15.155:0/2877958506 shutdown_connections 2024-03-25T18:24:58.372 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:58.370+0000 7f1a9ec57640 1 -- 172.21.15.155:0/2877958506 >> 172.21.15.155:0/2877958506 conn(0x7f1a980b5c00 msgr2=0x7f1a9813b990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:58.372 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:58.370+0000 7f1a9ec57640 1 -- 172.21.15.155:0/2877958506 shutdown_connections 2024-03-25T18:24:58.372 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:58.370+0000 7f1a9ec57640 1 -- 172.21.15.155:0/2877958506 wait complete. 2024-03-25T18:24:58.372 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:24:58.370+0000 7f1a9ec57640 1 librados: shutdown 2024-03-25T18:24:58.381 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:58.385 INFO:tasks.ceph.osd.6.smithi167.stderr:==00:00:00:09.754 35185== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:58.397 INFO:tasks.ceph.osd.2.smithi155.stderr:==00:00:00:09.779 35523== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:58.409 INFO:tasks.ceph.osd.7.smithi167.stderr:==00:00:00:09.776 35186== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:58.494 INFO:tasks.ceph.osd.0.smithi155.stderr:==00:00:00:09.865 35524== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:58.507 INFO:tasks.ceph.osd.5.smithi167.stderr:==00:00:00:09.880 35184== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:58.568 INFO:tasks.ceph.osd.3.smithi155.stderr:==00:00:00:09.927 35526== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:58.582 INFO:tasks.ceph.osd.4.smithi167.stderr:==00:00:00:09.937 35187== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:58.595 INFO:tasks.ceph.osd.1.smithi155.stderr:==00:00:00:09.961 35525== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:58.662 INFO:tasks.ceph.osd.2.smithi155.stderr:==00:00:00:10.044 35523== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:58.677 INFO:tasks.ceph.osd.6.smithi167.stderr:==00:00:00:10.046 35185== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:58.681 INFO:tasks.ceph.osd.7.smithi167.stderr:==00:00:00:10.048 35186== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:58.784 INFO:tasks.ceph.osd.0.smithi155.stderr:==00:00:00:10.156 35524== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:58.808 INFO:tasks.ceph.osd.5.smithi167.stderr:==00:00:00:10.181 35184== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:58.852 INFO:tasks.ceph.osd.4.smithi167.stderr:==00:00:00:10.207 35187== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:58.856 INFO:tasks.ceph.osd.3.smithi155.stderr:==00:00:00:10.215 35526== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:58.889 INFO:tasks.ceph.osd.1.smithi155.stderr:==00:00:00:10.254 35525== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:58.929 INFO:tasks.ceph.osd.2.smithi155.stderr:==00:00:00:10.310 35523== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:59.028 INFO:tasks.ceph.osd.7.smithi167.stderr:==00:00:00:10.312 35186== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:59.028 INFO:tasks.ceph.osd.6.smithi167.stderr:==00:00:00:10.339 35185== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:59.084 INFO:tasks.ceph.osd.0.smithi155.stderr:==00:00:00:10.455 35524== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:59.095 INFO:tasks.ceph.osd.5.smithi167.stderr:==00:00:00:10.468 35184== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:59.117 INFO:tasks.ceph.osd.4.smithi167.stderr:==00:00:00:10.472 35187== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:59.146 INFO:tasks.ceph.osd.3.smithi155.stderr:==00:00:00:10.506 35526== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:59.179 INFO:tasks.ceph.osd.1.smithi155.stderr:==00:00:00:10.545 35525== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:59.209 INFO:tasks.ceph.osd.2.smithi155.stderr:==00:00:00:10.590 35523== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:59.373 INFO:tasks.ceph.osd.0.smithi155.stderr:==00:00:00:10.744 35524== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:59.384 INFO:tasks.ceph.osd.4.smithi167.stderr:==00:00:00:10.739 35187== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:59.419 INFO:tasks.ceph.osd.7.smithi167.stderr:==00:00:00:10.785 35186== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:59.441 INFO:tasks.ceph.osd.3.smithi155.stderr:==00:00:00:10.800 35526== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:59.473 INFO:tasks.ceph.osd.7.smithi167.stderr:==00:00:00:10.840 35186== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:59.475 INFO:tasks.ceph.osd.1.smithi155.stderr:==00:00:00:10.841 35525== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:59.492 INFO:tasks.ceph.osd.2.smithi155.stderr:==00:00:00:10.873 35523== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:59.492 INFO:tasks.ceph.osd.6.smithi167.stderr:==00:00:00:10.862 35185== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:59.546 INFO:tasks.ceph.osd.6.smithi167.stderr:==00:00:00:10.916 35185== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:59.618 INFO:tasks.ceph.osd.5.smithi167.stderr:==00:00:00:10.991 35184== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:59.659 INFO:tasks.ceph.osd.0.smithi155.stderr:==00:00:00:11.030 35524== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:59.672 INFO:tasks.ceph.osd.5.smithi167.stderr:==00:00:00:11.045 35184== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:59.733 INFO:tasks.ceph.osd.3.smithi155.stderr:==00:00:00:11.092 35526== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:59.860 INFO:tasks.ceph.osd.4.smithi167.stderr:==00:00:00:11.215 35187== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:59.915 INFO:tasks.ceph.osd.4.smithi167.stderr:==00:00:00:11.270 35187== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:59.962 INFO:tasks.ceph.osd.2.smithi155.stderr:==00:00:00:11.343 35523== Warning: unimplemented fcntl command: 1038 2024-03-25T18:24:59.995 INFO:tasks.ceph.osd.1.smithi155.stderr:==00:00:00:11.360 35525== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:00.045 INFO:tasks.ceph.osd.2.smithi155.stderr:==00:00:00:11.427 35523== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:00.049 INFO:tasks.ceph.osd.1.smithi155.stderr:==00:00:00:11.415 35525== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:00.208 INFO:tasks.ceph.osd.0.smithi155.stderr:==00:00:00:11.580 35524== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:00.261 INFO:tasks.ceph.osd.3.smithi155.stderr:==00:00:00:11.620 35526== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:00.262 INFO:tasks.ceph.osd.0.smithi155.stderr:==00:00:00:11.634 35524== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:00.316 INFO:tasks.ceph.osd.3.smithi155.stderr:==00:00:00:11.675 35526== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:04.862 INFO:tasks.ceph.osd.6.smithi167.stderr:==00:00:00:16.232 35185== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:04.964 INFO:tasks.ceph.osd.7.smithi167.stderr:==00:00:00:16.331 35186== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:05.033 INFO:tasks.ceph.osd.5.smithi167.stderr:==00:00:00:16.406 35184== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:05.188 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:05.215 INFO:tasks.ceph.osd.4.smithi167.stderr:==00:00:00:16.570 35187== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:05.295 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.294+0000 7f21554b3640 10 monclient: get_monmap_and_config 2024-03-25T18:25:05.295 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.294+0000 7f21554b3640 10 monclient: build_initial_monmap 2024-03-25T18:25:05.295 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.294+0000 7f21554b3640 10 monclient: monmap: 2024-03-25T18:25:05.295 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:25:05.295 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:25:05.295 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:25:05.295341+0000 2024-03-25T18:25:05.296 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:25:05.295341+0000 2024-03-25T18:25:05.296 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:05.296 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:25:05.296 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:25:05.296 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:25:05.296 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:25:05.296 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:05.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.294+0000 7f21554b3640 1 Processor -- start 2024-03-25T18:25:05.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.295+0000 7f21554b3640 1 -- start start 2024-03-25T18:25:05.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.295+0000 7f21554b3640 10 monclient: init 2024-03-25T18:25:05.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.295+0000 7f21554b3640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:05.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.295+0000 7f21554b3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:25:05.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.295+0000 7f21554b3640 10 monclient(hunting): picked mon.noname-b con 0x7f21500b4d60 addr v1:172.21.15.155:6790/0 2024-03-25T18:25:05.299 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.295+0000 7f21554b3640 10 monclient(hunting): picked mon.noname-c con 0x7f21500b1730 addr v1:172.21.15.167:6789/0 2024-03-25T18:25:05.299 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.295+0000 7f21554b3640 10 monclient(hunting): picked mon.noname-a con 0x7f215014be90 addr v1:172.21.15.155:6789/0 2024-03-25T18:25:05.299 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.295+0000 7f21554b3640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21500af940 con 0x7f215014be90 2024-03-25T18:25:05.299 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.295+0000 7f21554b3640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21500b0a70 con 0x7f21500b1730 2024-03-25T18:25:05.299 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.295+0000 7f21554b3640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21500b0d50 con 0x7f21500b4d60 2024-03-25T18:25:05.299 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.295+0000 7f21554b3640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:05.299 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.295+0000 7f21554b3640 10 monclient(hunting): authenticate will time out at 7551.605599s 2024-03-25T18:25:05.299 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.296+0000 7f214effd640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f21500b4d60 0x7f215013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:60244/0 (socket says 172.21.15.155:60244) 2024-03-25T18:25:05.299 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.296+0000 7f214effd640 1 -- 172.21.15.155:0/1151718316 learned_addr learned my addr 172.21.15.155:0/1151718316 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:25:05.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.298+0000 7f214dffb640 1 -- 172.21.15.155:0/1151718316 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2513667780 0 0) 0x7f21500b0d50 con 0x7f21500b4d60 2024-03-25T18:25:05.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.298+0000 7f214dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:25:05.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.298+0000 7f214dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:05.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.298+0000 7f214dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:05.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.298+0000 7f214dffb640 10 monclient(hunting): my global_id is 4211 2024-03-25T18:25:05.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.298+0000 7f214dffb640 1 -- 172.21.15.155:0/1151718316 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2138003520 con 0x7f21500b4d60 2024-03-25T18:25:05.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.299+0000 7f214dffb640 1 -- 172.21.15.155:0/1151718316 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3524029479 0 0) 0x7f21500af940 con 0x7f215014be90 2024-03-25T18:25:05.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.299+0000 7f214dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:25:05.302 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.299+0000 7f214dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:05.302 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.299+0000 7f214dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:05.302 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.299+0000 7f214dffb640 10 monclient(hunting): my global_id is 4209 2024-03-25T18:25:05.302 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.299+0000 7f214dffb640 1 -- 172.21.15.155:0/1151718316 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21500b0d50 con 0x7f215014be90 2024-03-25T18:25:05.302 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.299+0000 7f214dffb640 1 -- 172.21.15.155:0/1151718316 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3520957035 0 0) 0x7f21500b0a70 con 0x7f21500b1730 2024-03-25T18:25:05.302 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.299+0000 7f214dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:25:05.302 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.299+0000 7f214dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:05.302 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.299+0000 7f214dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:05.302 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.299+0000 7f214dffb640 10 monclient(hunting): my global_id is 4210 2024-03-25T18:25:05.302 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.299+0000 7f214dffb640 1 -- 172.21.15.155:0/1151718316 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21500af940 con 0x7f21500b1730 2024-03-25T18:25:05.310 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.302+0000 7f214dffb640 1 -- 172.21.15.155:0/1151718316 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3685747376 0 0) 0x7f21500b0d50 con 0x7f215014be90 2024-03-25T18:25:05.310 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.302+0000 7f214dffb640 1 -- 172.21.15.155:0/1151718316 >> v1:172.21.15.155:6790/0 conn(0x7f21500b4d60 legacy=0x7f215013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:05.310 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.303+0000 7f214dffb640 1 -- 172.21.15.155:0/1151718316 >> v1:172.21.15.167:6789/0 conn(0x7f21500b1730 legacy=0x7f21500b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:05.310 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.303+0000 7f214dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:25:05.311 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.303+0000 7f214dffb640 1 monclient: found mon.noname-a 2024-03-25T18:25:05.311 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.303+0000 7f214dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:05.311 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.303+0000 7f214dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-25T18:25:05.311 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.303+0000 7f214dffb640 1 -- 172.21.15.155:0/1151718316 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21500b1030 con 0x7f215014be90 2024-03-25T18:25:05.311 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.303+0000 7f214dffb640 10 monclient: _finish_auth 0 2024-03-25T18:25:05.311 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.303+0000 7f214dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:25:05.311 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.303+0000 7f214dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:05.311 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.303+0000 7f214dffb640 1 -- 172.21.15.155:0/1151718316 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f21480019f0 con 0x7f215014be90 2024-03-25T18:25:05.311 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.303+0000 7f214dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:05.311 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.303+0000 7f214dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:05.311 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.303+0000 7f214dffb640 10 monclient: dump: 2024-03-25T18:25:05.311 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:25:05.311 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:25:05.311 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:05.312 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:05.312 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:05.312 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:25:05.312 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:25:05.312 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:25:05.312 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:25:05.312 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:05.312 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.303+0000 7f21554b3640 5 monclient: authenticate success, global_id 4209 2024-03-25T18:25:05.312 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.303+0000 7f21554b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:05.312 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.305+0000 7f214dffb640 1 -- 172.21.15.155:0/1151718316 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2148001b50 con 0x7f215014be90 2024-03-25T18:25:05.312 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.305+0000 7f214dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:05.312 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.305+0000 7f214dffb640 1 -- 172.21.15.155:0/1151718316 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f2148001fa0 con 0x7f215014be90 2024-03-25T18:25:05.312 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.305+0000 7f21554b3640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:05.312 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.305+0000 7f21554b3640 4 set_mon_vals no callback set 2024-03-25T18:25:05.312 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.305+0000 7f214dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:05.312 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.305+0000 7f214dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:05.313 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.305+0000 7f214dffb640 10 monclient: dump: 2024-03-25T18:25:05.313 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:25:05.313 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:25:05.313 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:05.313 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:05.313 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:05.313 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:25:05.313 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:25:05.313 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:25:05.313 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:25:05.313 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:05.313 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.305+0000 7f21554b3640 10 monclient: shutdown 2024-03-25T18:25:05.313 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.305+0000 7f21554b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:05.313 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.305+0000 7f21554b3640 1 -- 172.21.15.155:0/1151718316 >> v1:172.21.15.155:6789/0 conn(0x7f215014be90 legacy=0x7f215014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:05.313 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.305+0000 7f21554b3640 1 -- 172.21.15.155:0/1151718316 shutdown_connections 2024-03-25T18:25:05.314 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.305+0000 7f21554b3640 1 -- 172.21.15.155:0/1151718316 >> 172.21.15.155:0/1151718316 conn(0x7f21500617a0 msgr2=0x7f2150061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:05.314 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.305+0000 7f21554b3640 1 -- 172.21.15.155:0/1151718316 shutdown_connections 2024-03-25T18:25:05.314 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.305+0000 7f21554b3640 1 -- 172.21.15.155:0/1151718316 wait complete. 2024-03-25T18:25:05.314 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.306+0000 7f21554b3640 10 monclient: build_initial_monmap 2024-03-25T18:25:05.314 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.306+0000 7f21554b3640 10 monclient: monmap: 2024-03-25T18:25:05.314 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:25:05.314 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:05.314 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:25:05.314 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:25:05.314 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:05.314 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:25:05.314 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:25:05.314 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:25:05.314 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:25:05.315 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:05.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.306+0000 7f21554b3640 1 librados: starting msgr at 2024-03-25T18:25:05.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.306+0000 7f21554b3640 1 librados: starting objecter 2024-03-25T18:25:05.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.307+0000 7f21554b3640 1 Processor -- start 2024-03-25T18:25:05.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.307+0000 7f21554b3640 1 -- start start 2024-03-25T18:25:05.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.307+0000 7f21554b3640 1 librados: setting wanted keys 2024-03-25T18:25:05.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.307+0000 7f21554b3640 1 librados: calling monclient init 2024-03-25T18:25:05.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.307+0000 7f21554b3640 10 monclient: init 2024-03-25T18:25:05.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.307+0000 7f21554b3640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:05.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.307+0000 7f21554b3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:25:05.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.307+0000 7f21554b3640 10 monclient(hunting): picked mon.noname-a con 0x7f21500b1730 addr v1:172.21.15.155:6789/0 2024-03-25T18:25:05.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.307+0000 7f21554b3640 10 monclient(hunting): picked mon.noname-c con 0x7f21500b4d60 addr v1:172.21.15.155:6790/0 2024-03-25T18:25:05.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.307+0000 7f21554b3640 10 monclient(hunting): picked mon.noname-b con 0x7f215014be90 addr v1:172.21.15.167:6789/0 2024-03-25T18:25:05.316 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.307+0000 7f21554b3640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f215013bb90 con 0x7f21500b1730 2024-03-25T18:25:05.316 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.308+0000 7f21554b3640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21501dbc40 con 0x7f215014be90 2024-03-25T18:25:05.316 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.308+0000 7f21554b3640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2150141500 con 0x7f21500b4d60 2024-03-25T18:25:05.316 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.308+0000 7f21554b3640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:05.316 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.308+0000 7f21554b3640 10 monclient(hunting): authenticate will time out at 7551.617936s 2024-03-25T18:25:05.316 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.308+0000 7f214effd640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f21500b1730 0x7f215013a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:41546/0 (socket says 172.21.15.155:41546) 2024-03-25T18:25:05.316 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.308+0000 7f214effd640 1 -- 172.21.15.155:0/1991667711 learned_addr learned my addr 172.21.15.155:0/1991667711 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:25:05.316 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.308+0000 7f21377fe640 10 client.?.objecter ms_handle_connect 0x7f215014be90 2024-03-25T18:25:05.316 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.308+0000 7f21377fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:05.316 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.308+0000 7f21377fe640 10 client.?.objecter ms_handle_connect 0x7f21500b1730 2024-03-25T18:25:05.316 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.308+0000 7f21377fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:05.316 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.309+0000 7f21377fe640 10 client.?.objecter ms_handle_connect 0x7f21500b4d60 2024-03-25T18:25:05.316 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.309+0000 7f21377fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:05.320 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.310+0000 7f21377fe640 1 -- 172.21.15.155:0/1991667711 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3444595846 0 0) 0x7f2150141500 con 0x7f21500b4d60 2024-03-25T18:25:05.320 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.311+0000 7f21377fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:05.320 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.311+0000 7f21377fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:05.320 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.311+0000 7f21377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:05.320 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.311+0000 7f21377fe640 10 monclient(hunting): my global_id is 4214 2024-03-25T18:25:05.320 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.311+0000 7f21377fe640 1 -- 172.21.15.155:0/1991667711 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f213c004eb0 con 0x7f21500b4d60 2024-03-25T18:25:05.320 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.311+0000 7f21377fe640 1 -- 172.21.15.155:0/1991667711 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3884133243 0 0) 0x7f215013bb90 con 0x7f21500b1730 2024-03-25T18:25:05.320 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.311+0000 7f21377fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:05.320 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.311+0000 7f21377fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:05.320 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.311+0000 7f21377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:05.320 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.311+0000 7f21377fe640 10 monclient(hunting): my global_id is 4212 2024-03-25T18:25:05.320 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.311+0000 7f21377fe640 1 -- 172.21.15.155:0/1991667711 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2150141500 con 0x7f21500b1730 2024-03-25T18:25:05.320 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.311+0000 7f21377fe640 1 -- 172.21.15.155:0/1991667711 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2706955141 0 0) 0x7f21501dbc40 con 0x7f215014be90 2024-03-25T18:25:05.320 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.311+0000 7f21377fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:05.321 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.311+0000 7f21377fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:05.321 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.311+0000 7f21377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:05.321 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.311+0000 7f21377fe640 10 monclient(hunting): my global_id is 4213 2024-03-25T18:25:05.321 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.311+0000 7f21377fe640 1 -- 172.21.15.155:0/1991667711 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f213c005360 con 0x7f215014be90 2024-03-25T18:25:05.321 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.317+0000 7f21377fe640 1 -- 172.21.15.155:0/1991667711 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3574945218 0 0) 0x7f213c004eb0 con 0x7f21500b4d60 2024-03-25T18:25:05.321 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.317+0000 7f21377fe640 1 -- 172.21.15.155:0/1991667711 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f21501dbc40 con 0x7f21500b4d60 2024-03-25T18:25:05.321 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.317+0000 7f21377fe640 1 -- 172.21.15.155:0/1991667711 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f21440018f0 con 0x7f21500b4d60 2024-03-25T18:25:05.321 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.317+0000 7f21377fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:05.321 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.317+0000 7f21377fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:05.321 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.317+0000 7f21377fe640 10 monclient(hunting): dump: 2024-03-25T18:25:05.321 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:25:05.321 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:25:05.321 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:05.321 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:05.321 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:05.321 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:25:05.322 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:25:05.322 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:25:05.322 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:25:05.322 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:05.322 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.317+0000 7f21377fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:05.322 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.317+0000 7f21377fe640 1 -- 172.21.15.155:0/1991667711 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1953089964 0 0) 0x7f213c005360 con 0x7f215014be90 2024-03-25T18:25:05.322 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.317+0000 7f21377fe640 1 -- 172.21.15.155:0/1991667711 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f213c004eb0 con 0x7f215014be90 2024-03-25T18:25:05.322 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.318+0000 7f21377fe640 1 -- 172.21.15.155:0/1991667711 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2770950646 0 0) 0x7f2150141500 con 0x7f21500b1730 2024-03-25T18:25:05.322 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.318+0000 7f21377fe640 1 -- 172.21.15.155:0/1991667711 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f213c005360 con 0x7f21500b1730 2024-03-25T18:25:05.322 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.318+0000 7f21377fe640 1 -- 172.21.15.155:0/1991667711 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f21400013b0 con 0x7f21500b1730 2024-03-25T18:25:05.322 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.318+0000 7f21377fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:05.322 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.318+0000 7f21377fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:05.322 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.318+0000 7f21377fe640 10 monclient(hunting): dump: 2024-03-25T18:25:05.322 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:25:05.322 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:25:05.323 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:05.323 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:05.323 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:05.323 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:25:05.323 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:25:05.323 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:25:05.323 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:25:05.323 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:05.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.318+0000 7f21377fe640 1 -- 172.21.15.155:0/1991667711 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f2148001e60 con 0x7f215014be90 2024-03-25T18:25:05.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.318+0000 7f21377fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:05.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.318+0000 7f21377fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:05.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.318+0000 7f21377fe640 10 monclient(hunting): dump: 2024-03-25T18:25:05.323 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:25:05.323 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:25:05.323 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:05.323 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:05.323 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:05.324 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:25:05.324 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:25:05.324 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:25:05.324 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:25:05.324 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:05.326 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.322+0000 7f21377fe640 1 -- 172.21.15.155:0/1991667711 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 351950893 0 0) 0x7f21501dbc40 con 0x7f21500b4d60 2024-03-25T18:25:05.326 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.322+0000 7f21377fe640 1 -- 172.21.15.155:0/1991667711 >> v1:172.21.15.167:6789/0 conn(0x7f215014be90 legacy=0x7f2150071570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:05.326 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.322+0000 7f21377fe640 1 -- 172.21.15.155:0/1991667711 >> v1:172.21.15.155:6789/0 conn(0x7f21500b1730 legacy=0x7f215013a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:05.326 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.322+0000 7f21377fe640 10 monclient: _finish_hunting 0 2024-03-25T18:25:05.326 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.322+0000 7f21377fe640 1 monclient: found mon.c 2024-03-25T18:25:05.326 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.322+0000 7f21377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:05.326 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.322+0000 7f21377fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:25:05.327 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.322+0000 7f21377fe640 1 -- 172.21.15.155:0/1991667711 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2150141690 con 0x7f21500b4d60 2024-03-25T18:25:05.327 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.322+0000 7f21377fe640 10 monclient: _finish_auth 0 2024-03-25T18:25:05.327 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.322+0000 7f21377fe640 10 monclient: _check_auth_tickets 2024-03-25T18:25:05.327 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.322+0000 7f21377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:05.327 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.322+0000 7f21554b3640 5 monclient: authenticate success, global_id 4214 2024-03-25T18:25:05.327 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.322+0000 7f21554b3640 10 monclient: _renew_subs 2024-03-25T18:25:05.327 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.322+0000 7f21554b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:25:05.327 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.322+0000 7f21554b3640 1 -- 172.21.15.155:0/1991667711 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f215013aac0 con 0x7f21500b4d60 2024-03-25T18:25:05.327 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.322+0000 7f21554b3640 10 client.4214.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:05.327 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.322+0000 7f21554b3640 10 monclient: _renew_subs 2024-03-25T18:25:05.327 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.322+0000 7f21554b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:25:05.327 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.322+0000 7f21554b3640 1 -- 172.21.15.155:0/1991667711 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f21501399f0 con 0x7f21500b4d60 2024-03-25T18:25:05.327 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.322+0000 7f21554b3640 1 librados: init done 2024-03-25T18:25:05.327 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.323+0000 7f21554b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:05.327 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.323+0000 7f21554b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:05.328 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.323+0000 7f21554b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:25:05.328 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.323+0000 7f21554b3640 1 -- 172.21.15.155:0/1991667711 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21501dbe30 con 0x7f21500b4d60 2024-03-25T18:25:05.579 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.578+0000 7f21377fe640 1 -- 172.21.15.155:0/1991667711 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2144003610 con 0x7f21500b4d60 2024-03-25T18:25:05.579 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.578+0000 7f21377fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:05.579 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.578+0000 7f21377fe640 1 -- 172.21.15.155:0/1991667711 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f2144003920 con 0x7f21500b4d60 2024-03-25T18:25:05.579 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.578+0000 7f21377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:05.579 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.578+0000 7f21377fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:05.579 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.578+0000 7f21377fe640 10 monclient: dump: 2024-03-25T18:25:05.579 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:25:05.579 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:25:05.579 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:05.579 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:05.580 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:05.580 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:25:05.580 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:25:05.580 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:25:05.580 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:25:05.580 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:05.580 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.578+0000 7f214d7fa640 4 set_mon_vals no callback set 2024-03-25T18:25:05.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.579+0000 7f21377fe640 1 -- 172.21.15.155:0/1991667711 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7f21440198f0 con 0x7f21500b4d60 2024-03-25T18:25:05.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.579+0000 7f214effd640 10 monclient: get_auth_request con 0x7f213c06e510 auth_method 0 2024-03-25T18:25:05.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.580+0000 7f21377fe640 10 client.4214.objecter ms_handle_connect 0x7f213c06e510 2024-03-25T18:25:05.587 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.586+0000 7f21377fe640 1 -- 172.21.15.155:0/1991667711 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7f2144053510 con 0x7f21500b4d60 2024-03-25T18:25:05.587 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.586+0000 7f21377fe640 10 client.4214.objecter ms_dispatch 0x7f2150000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:05.587 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.586+0000 7f21377fe640 3 client.4214.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:05.587 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.586+0000 7f21377fe640 3 client.4214.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:05.587 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.586+0000 7f21377fe640 20 client.4214.objecter dump_active .. 0 homeless 2024-03-25T18:25:05.587 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.586+0000 7f21377fe640 1 -- 172.21.15.155:0/1991667711 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f214404f020 con 0x7f21500b4d60 2024-03-25T18:25:05.587 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.586+0000 7f21377fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:05.587 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.586+0000 7f21377fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:05.756 INFO:tasks.ceph.osd.2.smithi155.stderr:==00:00:00:17.137 35523== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:05.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.765+0000 7f21554b3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:05.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.765+0000 7f21554b3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:05.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.765+0000 7f21554b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:25:05.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.765+0000 7f21554b3640 1 -- 172.21.15.155:0/1991667711 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21500627b0 con 0x7f21500b4d60 2024-03-25T18:25:05.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.780+0000 7f21377fe640 1 -- 172.21.15.155:0/1991667711 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7f2144050f50 con 0x7f21500b4d60 2024-03-25T18:25:05.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.780+0000 7f21377fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:05.781 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.780+0000 7f21377fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:05.781 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:25:05.781 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:05.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.782+0000 7f21554b3640 10 librados: watch_flush enter 2024-03-25T18:25:05.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.782+0000 7f21554b3640 10 librados: watch_flush exit 2024-03-25T18:25:05.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.782+0000 7f21554b3640 20 client.4214.objecter shutdown clearing up homeless session... 2024-03-25T18:25:05.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.782+0000 7f21554b3640 10 client.4214.objecter successfully canceled tick 2024-03-25T18:25:05.785 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.783+0000 7f21554b3640 1 -- 172.21.15.155:0/1991667711 >> v1:172.21.15.167:6800/2919070503 conn(0x7f213c06e510 legacy=0x7f213c0709d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:05.785 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.783+0000 7f21554b3640 10 monclient: shutdown 2024-03-25T18:25:05.785 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.783+0000 7f21554b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:05.785 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.783+0000 7f21554b3640 1 -- 172.21.15.155:0/1991667711 >> v1:172.21.15.155:6790/0 conn(0x7f21500b4d60 legacy=0x7f21501dc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:05.785 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.783+0000 7f21554b3640 1 -- 172.21.15.155:0/1991667711 shutdown_connections 2024-03-25T18:25:05.785 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.783+0000 7f21554b3640 1 -- 172.21.15.155:0/1991667711 >> 172.21.15.155:0/1991667711 conn(0x7f21500617a0 msgr2=0x7f21501373b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:05.785 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.783+0000 7f21554b3640 1 -- 172.21.15.155:0/1991667711 shutdown_connections 2024-03-25T18:25:05.785 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.783+0000 7f21554b3640 1 -- 172.21.15.155:0/1991667711 wait complete. 2024-03-25T18:25:05.785 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:05.783+0000 7f21554b3640 1 librados: shutdown 2024-03-25T18:25:05.797 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:06.070 INFO:tasks.ceph.osd.1.smithi155.stderr:==00:00:00:17.436 35525== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:06.185 INFO:tasks.ceph.osd.3.smithi155.stderr:==00:00:00:17.544 35526== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:06.355 INFO:tasks.ceph.osd.0.smithi155.stderr:==00:00:00:17.727 35524== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:10.652 INFO:tasks.ceph.osd.7.smithi167.stderr:==00:00:00:22.019 35186== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:10.656 INFO:tasks.ceph.osd.7.smithi167.stderr:==00:00:00:22.023 35186== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:10.807 INFO:tasks.ceph.osd.6.smithi167.stderr:==00:00:00:22.177 35185== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:10.813 INFO:tasks.ceph.osd.6.smithi167.stderr:==00:00:00:22.183 35185== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:10.882 INFO:tasks.ceph.osd.5.smithi167.stderr:==00:00:00:22.255 35184== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:10.886 INFO:tasks.ceph.osd.5.smithi167.stderr:==00:00:00:22.259 35184== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:10.957 INFO:tasks.ceph.osd.4.smithi167.stderr:==00:00:00:22.311 35187== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:10.963 INFO:tasks.ceph.osd.4.smithi167.stderr:==00:00:00:22.317 35187== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:11.718 INFO:tasks.ceph.osd.2.smithi155.stderr:==00:00:00:23.099 35523== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:11.722 INFO:tasks.ceph.osd.2.smithi155.stderr:==00:00:00:23.103 35523== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:11.934 INFO:tasks.ceph.osd.1.smithi155.stderr:==00:00:00:23.299 35525== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:11.938 INFO:tasks.ceph.osd.1.smithi155.stderr:==00:00:00:23.303 35525== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:12.129 INFO:tasks.ceph.osd.3.smithi155.stderr:==00:00:00:23.488 35526== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:12.135 INFO:tasks.ceph.osd.3.smithi155.stderr:==00:00:00:23.494 35526== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:12.331 INFO:tasks.ceph.osd.0.smithi155.stderr:==00:00:00:23.703 35524== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:12.336 INFO:tasks.ceph.osd.0.smithi155.stderr:==00:00:00:23.708 35524== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:12.605 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:12.723 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.722+0000 7f0f2a1d6640 10 monclient: get_monmap_and_config 2024-03-25T18:25:12.723 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.722+0000 7f0f2a1d6640 10 monclient: build_initial_monmap 2024-03-25T18:25:12.723 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.722+0000 7f0f2a1d6640 10 monclient: monmap: 2024-03-25T18:25:12.724 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:25:12.724 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:25:12.724 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:25:12.723274+0000 2024-03-25T18:25:12.724 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:25:12.723274+0000 2024-03-25T18:25:12.724 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:12.724 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:25:12.724 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:25:12.724 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:25:12.724 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:25:12.724 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:12.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.723+0000 7f0f2a1d6640 1 Processor -- start 2024-03-25T18:25:12.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.723+0000 7f0f2a1d6640 1 -- start start 2024-03-25T18:25:12.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.723+0000 7f0f2a1d6640 10 monclient: init 2024-03-25T18:25:12.726 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.723+0000 7f0f2a1d6640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:12.726 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.723+0000 7f0f2a1d6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:25:12.726 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.723+0000 7f0f2a1d6640 10 monclient(hunting): picked mon.noname-a con 0x7f0f240a98f0 addr v1:172.21.15.155:6789/0 2024-03-25T18:25:12.726 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.724+0000 7f0f2a1d6640 10 monclient(hunting): picked mon.noname-b con 0x7f0f2413f710 addr v1:172.21.15.155:6790/0 2024-03-25T18:25:12.726 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.724+0000 7f0f2a1d6640 10 monclient(hunting): picked mon.noname-c con 0x7f0f241430c0 addr v1:172.21.15.167:6789/0 2024-03-25T18:25:12.726 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.724+0000 7f0f2a1d6640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f24142520 con 0x7f0f240a98f0 2024-03-25T18:25:12.726 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.724+0000 7f0f2a1d6640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f240b28a0 con 0x7f0f241430c0 2024-03-25T18:25:12.726 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.724+0000 7f0f2a1d6640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f240b2a30 con 0x7f0f2413f710 2024-03-25T18:25:12.726 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.724+0000 7f0f2a1d6640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:12.726 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.724+0000 7f0f2a1d6640 10 monclient(hunting): authenticate will time out at 7559.034039s 2024-03-25T18:25:12.727 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.724+0000 7f0f22ffd640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f0f2413f710 0x7f0f241429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:57924/0 (socket says 172.21.15.155:57924) 2024-03-25T18:25:12.727 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.724+0000 7f0f22ffd640 1 -- 172.21.15.155:0/3862913559 learned_addr learned my addr 172.21.15.155:0/3862913559 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:25:12.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.728+0000 7f0f227fc640 1 -- 172.21.15.155:0/3862913559 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 846299792 0 0) 0x7f0f240b2a30 con 0x7f0f2413f710 2024-03-25T18:25:12.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.728+0000 7f0f227fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:12.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.728+0000 7f0f227fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:12.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.728+0000 7f0f227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:12.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.728+0000 7f0f227fc640 10 monclient(hunting): my global_id is 4217 2024-03-25T18:25:12.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.728+0000 7f0f227fc640 1 -- 172.21.15.155:0/3862913559 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f0c003520 con 0x7f0f2413f710 2024-03-25T18:25:12.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.728+0000 7f0f227fc640 1 -- 172.21.15.155:0/3862913559 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1181497133 0 0) 0x7f0f240b28a0 con 0x7f0f241430c0 2024-03-25T18:25:12.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.728+0000 7f0f227fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:12.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.728+0000 7f0f227fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:12.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.728+0000 7f0f227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:12.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.728+0000 7f0f227fc640 10 monclient(hunting): my global_id is 4216 2024-03-25T18:25:12.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.728+0000 7f0f227fc640 1 -- 172.21.15.155:0/3862913559 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f240b2a30 con 0x7f0f241430c0 2024-03-25T18:25:12.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.728+0000 7f0f227fc640 1 -- 172.21.15.155:0/3862913559 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1783131598 0 0) 0x7f0f24142520 con 0x7f0f240a98f0 2024-03-25T18:25:12.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.728+0000 7f0f227fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:12.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.728+0000 7f0f227fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:12.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.728+0000 7f0f227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:12.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.728+0000 7f0f227fc640 10 monclient(hunting): my global_id is 4215 2024-03-25T18:25:12.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.728+0000 7f0f227fc640 1 -- 172.21.15.155:0/3862913559 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f240b28a0 con 0x7f0f240a98f0 2024-03-25T18:25:12.734 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.732+0000 7f0f227fc640 1 -- 172.21.15.155:0/3862913559 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2438781438 0 0) 0x7f0f240b2a30 con 0x7f0f241430c0 2024-03-25T18:25:12.734 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.732+0000 7f0f227fc640 1 -- 172.21.15.155:0/3862913559 >> v1:172.21.15.155:6790/0 conn(0x7f0f2413f710 legacy=0x7f0f241429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:12.734 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.732+0000 7f0f227fc640 1 -- 172.21.15.155:0/3862913559 >> v1:172.21.15.155:6789/0 conn(0x7f0f240a98f0 legacy=0x7f0f240a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:12.734 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.732+0000 7f0f227fc640 10 monclient: _finish_hunting 0 2024-03-25T18:25:12.734 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.732+0000 7f0f227fc640 1 monclient: found mon.noname-c 2024-03-25T18:25:12.734 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.732+0000 7f0f227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:12.734 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.732+0000 7f0f227fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.167:6789/0 2024-03-25T18:25:12.735 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.732+0000 7f0f227fc640 1 -- 172.21.15.155:0/3862913559 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f240b44c0 con 0x7f0f241430c0 2024-03-25T18:25:12.735 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.733+0000 7f0f227fc640 10 monclient: _finish_auth 0 2024-03-25T18:25:12.735 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.733+0000 7f0f227fc640 10 monclient: _check_auth_tickets 2024-03-25T18:25:12.735 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.733+0000 7f0f227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:12.735 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.733+0000 7f0f2a1d6640 5 monclient: authenticate success, global_id 4216 2024-03-25T18:25:12.735 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.733+0000 7f0f2a1d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:12.735 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.733+0000 7f0f227fc640 1 -- 172.21.15.155:0/3862913559 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f0f1c0019b0 con 0x7f0f241430c0 2024-03-25T18:25:12.735 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.733+0000 7f0f227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:12.735 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.733+0000 7f0f227fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:12.735 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.733+0000 7f0f227fc640 10 monclient: dump: 2024-03-25T18:25:12.735 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:25:12.735 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:25:12.735 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:12.735 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:12.735 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:12.736 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:25:12.736 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:25:12.736 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:25:12.736 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:25:12.736 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:12.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.733+0000 7f0f2a1d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:12.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.735+0000 7f0f227fc640 1 -- 172.21.15.155:0/3862913559 <== mon.1 v1:172.21.15.167:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f1c001c00 con 0x7f0f241430c0 2024-03-25T18:25:12.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.735+0000 7f0f227fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:12.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.735+0000 7f0f227fc640 1 -- 172.21.15.155:0/3862913559 <== mon.1 v1:172.21.15.167:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f0f1c001f90 con 0x7f0f241430c0 2024-03-25T18:25:12.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.735+0000 7f0f227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:12.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.735+0000 7f0f227fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:12.737 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.735+0000 7f0f227fc640 10 monclient: dump: 2024-03-25T18:25:12.737 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:25:12.737 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:25:12.737 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:12.737 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:12.737 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:12.737 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:25:12.737 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:25:12.737 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:25:12.737 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:25:12.737 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:12.737 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.735+0000 7f0f2a1d6640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:12.737 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.735+0000 7f0f2a1d6640 4 set_mon_vals no callback set 2024-03-25T18:25:12.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.736+0000 7f0f2a1d6640 10 monclient: shutdown 2024-03-25T18:25:12.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.736+0000 7f0f2a1d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:12.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.736+0000 7f0f2a1d6640 1 -- 172.21.15.155:0/3862913559 >> v1:172.21.15.167:6789/0 conn(0x7f0f241430c0 legacy=0x7f0f2414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:12.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.736+0000 7f0f2a1d6640 1 -- 172.21.15.155:0/3862913559 shutdown_connections 2024-03-25T18:25:12.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.736+0000 7f0f2a1d6640 1 -- 172.21.15.155:0/3862913559 >> 172.21.15.155:0/3862913559 conn(0x7f0f240b7ea0 msgr2=0x7f0f240b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:12.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.736+0000 7f0f2a1d6640 1 -- 172.21.15.155:0/3862913559 shutdown_connections 2024-03-25T18:25:12.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.736+0000 7f0f2a1d6640 1 -- 172.21.15.155:0/3862913559 wait complete. 2024-03-25T18:25:12.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.737+0000 7f0f2a1d6640 10 monclient: build_initial_monmap 2024-03-25T18:25:12.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.737+0000 7f0f2a1d6640 10 monclient: monmap: 2024-03-25T18:25:12.739 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:25:12.739 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:12.739 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:25:12.739 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:25:12.739 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:12.739 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:25:12.740 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:25:12.740 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:25:12.740 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:25:12.740 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:12.740 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.738+0000 7f0f2a1d6640 1 librados: starting msgr at 2024-03-25T18:25:12.740 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.738+0000 7f0f2a1d6640 1 librados: starting objecter 2024-03-25T18:25:12.740 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.738+0000 7f0f2a1d6640 1 Processor -- start 2024-03-25T18:25:12.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.738+0000 7f0f2a1d6640 1 -- start start 2024-03-25T18:25:12.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.738+0000 7f0f2a1d6640 1 librados: setting wanted keys 2024-03-25T18:25:12.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.738+0000 7f0f2a1d6640 1 librados: calling monclient init 2024-03-25T18:25:12.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.738+0000 7f0f2a1d6640 10 monclient: init 2024-03-25T18:25:12.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.739+0000 7f0f2a1d6640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:12.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.739+0000 7f0f2a1d6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:25:12.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.739+0000 7f0f2a1d6640 10 monclient(hunting): picked mon.noname-a con 0x7f0f240a98f0 addr v1:172.21.15.155:6789/0 2024-03-25T18:25:12.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.739+0000 7f0f2a1d6640 10 monclient(hunting): picked mon.noname-c con 0x7f0f2413f710 addr v1:172.21.15.155:6790/0 2024-03-25T18:25:12.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.739+0000 7f0f2a1d6640 10 monclient(hunting): picked mon.noname-b con 0x7f0f241430c0 addr v1:172.21.15.167:6789/0 2024-03-25T18:25:12.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.739+0000 7f0f2a1d6640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f240abca0 con 0x7f0f240a98f0 2024-03-25T18:25:12.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.739+0000 7f0f2a1d6640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f241dad10 con 0x7f0f241430c0 2024-03-25T18:25:12.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.739+0000 7f0f2a1d6640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f24138c80 con 0x7f0f2413f710 2024-03-25T18:25:12.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.739+0000 7f0f2a1d6640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:12.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.739+0000 7f0f2a1d6640 10 monclient(hunting): authenticate will time out at 7559.049370s 2024-03-25T18:25:12.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.740+0000 7f0f22ffd640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f0f2413f710 0x7f0f241db230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:57928/0 (socket says 172.21.15.155:57928) 2024-03-25T18:25:12.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.740+0000 7f0f22ffd640 1 -- 172.21.15.155:0/3419092141 learned_addr learned my addr 172.21.15.155:0/3419092141 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:25:12.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.740+0000 7f0f0bfff640 10 client.?.objecter ms_handle_connect 0x7f0f2413f710 2024-03-25T18:25:12.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.740+0000 7f0f0bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:12.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.740+0000 7f0f0bfff640 10 client.?.objecter ms_handle_connect 0x7f0f240a98f0 2024-03-25T18:25:12.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.740+0000 7f0f0bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:12.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.740+0000 7f0f0bfff640 10 client.?.objecter ms_handle_connect 0x7f0f241430c0 2024-03-25T18:25:12.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.740+0000 7f0f0bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:12.744 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.743+0000 7f0f0bfff640 1 -- 172.21.15.155:0/3419092141 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2325296807 0 0) 0x7f0f24138c80 con 0x7f0f2413f710 2024-03-25T18:25:12.744 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.743+0000 7f0f0bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:25:12.744 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.743+0000 7f0f0bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:12.744 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.743+0000 7f0f0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:12.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.743+0000 7f0f0bfff640 10 monclient(hunting): my global_id is 4220 2024-03-25T18:25:12.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.743+0000 7f0f0bfff640 1 -- 172.21.15.155:0/3419092141 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f10004eb0 con 0x7f0f2413f710 2024-03-25T18:25:12.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.743+0000 7f0f0bfff640 1 -- 172.21.15.155:0/3419092141 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1099327241 0 0) 0x7f0f240abca0 con 0x7f0f240a98f0 2024-03-25T18:25:12.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.743+0000 7f0f0bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:25:12.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.743+0000 7f0f0bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:12.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.743+0000 7f0f0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:12.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.743+0000 7f0f0bfff640 10 monclient(hunting): my global_id is 4218 2024-03-25T18:25:12.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.743+0000 7f0f0bfff640 1 -- 172.21.15.155:0/3419092141 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f24138c80 con 0x7f0f240a98f0 2024-03-25T18:25:12.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.743+0000 7f0f0bfff640 1 -- 172.21.15.155:0/3419092141 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2230615640 0 0) 0x7f0f241dad10 con 0x7f0f241430c0 2024-03-25T18:25:12.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.743+0000 7f0f0bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:25:12.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.743+0000 7f0f0bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:12.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.743+0000 7f0f0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:12.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.743+0000 7f0f0bfff640 10 monclient(hunting): my global_id is 4219 2024-03-25T18:25:12.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.743+0000 7f0f0bfff640 1 -- 172.21.15.155:0/3419092141 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f240abca0 con 0x7f0f241430c0 2024-03-25T18:25:12.750 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.749+0000 7f0f0bfff640 1 -- 172.21.15.155:0/3419092141 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2146426094 0 0) 0x7f0f24138c80 con 0x7f0f240a98f0 2024-03-25T18:25:12.750 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.749+0000 7f0f0bfff640 1 -- 172.21.15.155:0/3419092141 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f241dad10 con 0x7f0f240a98f0 2024-03-25T18:25:12.750 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.749+0000 7f0f0bfff640 1 -- 172.21.15.155:0/3419092141 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f0f14001820 con 0x7f0f240a98f0 2024-03-25T18:25:12.750 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.749+0000 7f0f0bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:12.750 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.749+0000 7f0f0bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:12.750 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.749+0000 7f0f0bfff640 10 monclient(hunting): dump: 2024-03-25T18:25:12.750 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:25:12.750 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:25:12.751 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:12.751 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:12.751 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:12.751 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:25:12.751 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:25:12.751 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:25:12.751 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:25:12.751 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:12.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.749+0000 7f0f0bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:12.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.750+0000 7f0f0bfff640 1 -- 172.21.15.155:0/3419092141 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3719303580 0 0) 0x7f0f10004eb0 con 0x7f0f2413f710 2024-03-25T18:25:12.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.750+0000 7f0f0bfff640 1 -- 172.21.15.155:0/3419092141 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f24138c80 con 0x7f0f2413f710 2024-03-25T18:25:12.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.750+0000 7f0f0bfff640 1 -- 172.21.15.155:0/3419092141 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f0f18001830 con 0x7f0f2413f710 2024-03-25T18:25:12.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.750+0000 7f0f0bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:12.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.751+0000 7f0f0bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:12.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.751+0000 7f0f0bfff640 10 monclient(hunting): dump: 2024-03-25T18:25:12.752 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:25:12.752 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:25:12.752 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:12.753 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:12.753 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:12.753 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:25:12.753 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:25:12.753 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:25:12.753 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:25:12.753 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:12.754 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.753+0000 7f0f0bfff640 1 -- 172.21.15.155:0/3419092141 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2894918453 0 0) 0x7f0f240abca0 con 0x7f0f241430c0 2024-03-25T18:25:12.754 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.753+0000 7f0f0bfff640 1 -- 172.21.15.155:0/3419092141 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f10004eb0 con 0x7f0f241430c0 2024-03-25T18:25:12.754 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.753+0000 7f0f0bfff640 1 -- 172.21.15.155:0/3419092141 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f0f1c001f10 con 0x7f0f241430c0 2024-03-25T18:25:12.754 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.753+0000 7f0f0bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:12.754 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.753+0000 7f0f0bfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:12.754 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.753+0000 7f0f0bfff640 10 monclient(hunting): dump: 2024-03-25T18:25:12.755 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:25:12.755 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:25:12.755 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:12.755 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:12.755 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:12.755 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:25:12.755 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:25:12.755 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:25:12.755 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:25:12.755 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:12.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.754+0000 7f0f0bfff640 1 -- 172.21.15.155:0/3419092141 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 65242628 0 0) 0x7f0f24138c80 con 0x7f0f2413f710 2024-03-25T18:25:12.758 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.754+0000 7f0f0bfff640 1 -- 172.21.15.155:0/3419092141 >> v1:172.21.15.167:6789/0 conn(0x7f0f241430c0 legacy=0x7f0f241d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:12.758 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.754+0000 7f0f0bfff640 1 -- 172.21.15.155:0/3419092141 >> v1:172.21.15.155:6789/0 conn(0x7f0f240a98f0 legacy=0x7f0f24138570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:12.758 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.754+0000 7f0f0bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:25:12.758 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.754+0000 7f0f0bfff640 1 monclient: found mon.c 2024-03-25T18:25:12.758 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.754+0000 7f0f0bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:12.758 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.754+0000 7f0f0bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:25:12.758 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.754+0000 7f0f0bfff640 1 -- 172.21.15.155:0/3419092141 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f241db940 con 0x7f0f2413f710 2024-03-25T18:25:12.758 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.754+0000 7f0f0bfff640 10 monclient: _finish_auth 0 2024-03-25T18:25:12.758 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.754+0000 7f0f0bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:25:12.758 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.754+0000 7f0f0bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:12.758 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.754+0000 7f0f2a1d6640 5 monclient: authenticate success, global_id 4220 2024-03-25T18:25:12.758 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.754+0000 7f0f2a1d6640 10 monclient: _renew_subs 2024-03-25T18:25:12.758 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.754+0000 7f0f2a1d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:25:12.758 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.754+0000 7f0f2a1d6640 1 -- 172.21.15.155:0/3419092141 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f241d41e0 con 0x7f0f2413f710 2024-03-25T18:25:12.759 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.754+0000 7f0f2a1d6640 10 client.4220.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:12.759 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.754+0000 7f0f2a1d6640 10 monclient: _renew_subs 2024-03-25T18:25:12.759 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.754+0000 7f0f2a1d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:25:12.759 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.754+0000 7f0f2a1d6640 1 -- 172.21.15.155:0/3419092141 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f241d5150 con 0x7f0f2413f710 2024-03-25T18:25:12.759 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.754+0000 7f0f2a1d6640 1 librados: init done 2024-03-25T18:25:12.759 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.756+0000 7f0f2a1d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:12.759 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.756+0000 7f0f2a1d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:12.759 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.756+0000 7f0f2a1d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:25:12.759 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:12.756+0000 7f0f2a1d6640 1 -- 172.21.15.155:0/3419092141 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f240b3ef0 con 0x7f0f2413f710 2024-03-25T18:25:13.073 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:13.072+0000 7f0f0bfff640 1 -- 172.21.15.155:0/3419092141 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f18001990 con 0x7f0f2413f710 2024-03-25T18:25:13.074 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:13.072+0000 7f0f0bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:13.074 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:13.072+0000 7f0f0bfff640 1 -- 172.21.15.155:0/3419092141 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f0f18004980 con 0x7f0f2413f710 2024-03-25T18:25:13.074 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:13.072+0000 7f0f0bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:13.074 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:13.072+0000 7f0f0bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:13.074 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:13.072+0000 7f0f21ffb640 4 set_mon_vals no callback set 2024-03-25T18:25:13.074 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:13.073+0000 7f0f0bfff640 10 monclient: dump: 2024-03-25T18:25:13.074 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:25:13.074 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:25:13.074 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:13.074 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:13.074 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:13.074 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:25:13.074 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:25:13.075 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:25:13.075 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:25:13.075 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:13.075 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:13.073+0000 7f0f0bfff640 1 -- 172.21.15.155:0/3419092141 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7f0f1801aa10 con 0x7f0f2413f710 2024-03-25T18:25:13.075 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:13.074+0000 7f0f237fe640 10 monclient: get_auth_request con 0x7f0f1006e0c0 auth_method 0 2024-03-25T18:25:13.075 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:13.074+0000 7f0f0bfff640 10 client.4220.objecter ms_handle_connect 0x7f0f1006e0c0 2024-03-25T18:25:13.078 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:13.077+0000 7f0f0bfff640 1 -- 172.21.15.155:0/3419092141 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7f0f18054790 con 0x7f0f2413f710 2024-03-25T18:25:13.078 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:13.077+0000 7f0f0bfff640 10 client.4220.objecter ms_dispatch 0x7f0f24000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:13.078 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:13.077+0000 7f0f0bfff640 3 client.4220.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:13.078 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:13.077+0000 7f0f0bfff640 3 client.4220.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:13.078 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:13.077+0000 7f0f0bfff640 20 client.4220.objecter dump_active .. 0 homeless 2024-03-25T18:25:13.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:13.137+0000 7f0f0bfff640 1 -- 172.21.15.155:0/3419092141 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0f180500d0 con 0x7f0f2413f710 2024-03-25T18:25:13.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:13.137+0000 7f0f0bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:13.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:13.137+0000 7f0f0bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:13.461 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:13.461+0000 7f0f2a1d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:13.462 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:13.461+0000 7f0f2a1d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:13.462 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:13.461+0000 7f0f2a1d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:25:13.462 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:13.461+0000 7f0f2a1d6640 1 -- 172.21.15.155:0/3419092141 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0f241daea0 con 0x7f0f2413f710 2024-03-25T18:25:13.478 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:13.477+0000 7f0f0bfff640 1 -- 172.21.15.155:0/3419092141 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7f0f180521d0 con 0x7f0f2413f710 2024-03-25T18:25:13.478 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:13.477+0000 7f0f0bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:13.478 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:13.477+0000 7f0f0bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:13.478 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:25:13.478 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:13.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:13.479+0000 7f0f2a1d6640 10 librados: watch_flush enter 2024-03-25T18:25:13.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:13.479+0000 7f0f2a1d6640 10 librados: watch_flush exit 2024-03-25T18:25:13.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:13.480+0000 7f0f2a1d6640 20 client.4220.objecter shutdown clearing up homeless session... 2024-03-25T18:25:13.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:13.480+0000 7f0f2a1d6640 10 client.4220.objecter successfully canceled tick 2024-03-25T18:25:13.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:13.480+0000 7f0f2a1d6640 1 -- 172.21.15.155:0/3419092141 >> v1:172.21.15.167:6800/2919070503 conn(0x7f0f1006e0c0 legacy=0x7f0f10070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:13.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:13.480+0000 7f0f2a1d6640 10 monclient: shutdown 2024-03-25T18:25:13.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:13.480+0000 7f0f2a1d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:13.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:13.480+0000 7f0f2a1d6640 1 -- 172.21.15.155:0/3419092141 >> v1:172.21.15.155:6790/0 conn(0x7f0f2413f710 legacy=0x7f0f241db230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:13.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:13.480+0000 7f0f2a1d6640 1 -- 172.21.15.155:0/3419092141 shutdown_connections 2024-03-25T18:25:13.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:13.480+0000 7f0f2a1d6640 1 -- 172.21.15.155:0/3419092141 >> 172.21.15.155:0/3419092141 conn(0x7f0f240b7ea0 msgr2=0x7f0f24074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:13.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:13.480+0000 7f0f2a1d6640 1 -- 172.21.15.155:0/3419092141 shutdown_connections 2024-03-25T18:25:13.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:13.481+0000 7f0f2a1d6640 1 -- 172.21.15.155:0/3419092141 wait complete. 2024-03-25T18:25:13.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:13.481+0000 7f0f2a1d6640 1 librados: shutdown 2024-03-25T18:25:13.495 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:14.861 INFO:tasks.ceph.osd.6.smithi167.stderr:==00:00:00:26.231 35185== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:14.866 INFO:tasks.ceph.osd.7.smithi167.stderr:==00:00:00:26.233 35186== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:15.092 INFO:tasks.ceph.osd.5.smithi167.stderr:==00:00:00:26.465 35184== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:15.430 INFO:tasks.ceph.osd.4.smithi167.stderr:==00:00:00:26.785 35187== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:15.828 INFO:tasks.ceph.osd.2.smithi155.stderr:==00:00:00:27.209 35523== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:16.454 INFO:tasks.ceph.osd.3.smithi155.stderr:==00:00:00:27.813 35526== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:16.497 INFO:tasks.ceph.osd.1.smithi155.stderr:==00:00:00:27.862 35525== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:16.776 INFO:tasks.ceph.osd.0.smithi155.stderr:==00:00:00:28.148 35524== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:20.302 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:20.391 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.390+0000 7fac0de0f640 10 monclient: get_monmap_and_config 2024-03-25T18:25:20.391 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.390+0000 7fac0de0f640 10 monclient: build_initial_monmap 2024-03-25T18:25:20.391 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.390+0000 7fac0de0f640 10 monclient: monmap: 2024-03-25T18:25:20.391 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:25:20.391 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:25:20.391 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:25:20.391381+0000 2024-03-25T18:25:20.391 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:25:20.391381+0000 2024-03-25T18:25:20.392 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:20.392 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:25:20.392 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:25:20.392 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:25:20.392 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:25:20.392 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:20.393 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.391+0000 7fac0de0f640 1 Processor -- start 2024-03-25T18:25:20.393 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.391+0000 7fac0de0f640 1 -- start start 2024-03-25T18:25:20.393 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.391+0000 7fac0de0f640 10 monclient: init 2024-03-25T18:25:20.394 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.391+0000 7fac0de0f640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:20.394 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.391+0000 7fac0de0f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:25:20.394 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.391+0000 7fac0de0f640 10 monclient(hunting): picked mon.noname-c con 0x7fac080b4d60 addr v1:172.21.15.167:6789/0 2024-03-25T18:25:20.394 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.391+0000 7fac0de0f640 10 monclient(hunting): picked mon.noname-b con 0x7fac080b1730 addr v1:172.21.15.155:6790/0 2024-03-25T18:25:20.394 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.391+0000 7fac0de0f640 10 monclient(hunting): picked mon.noname-a con 0x7fac0814be90 addr v1:172.21.15.155:6789/0 2024-03-25T18:25:20.394 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.391+0000 7fac0de0f640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac080af940 con 0x7fac0814be90 2024-03-25T18:25:20.394 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.391+0000 7fac0de0f640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac080b0a70 con 0x7fac080b4d60 2024-03-25T18:25:20.394 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.391+0000 7fac0de0f640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac080b0d50 con 0x7fac080b1730 2024-03-25T18:25:20.394 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.391+0000 7fac0de0f640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:20.395 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.391+0000 7fac0de0f640 10 monclient(hunting): authenticate will time out at 7566.701649s 2024-03-25T18:25:20.395 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.392+0000 7fac07fff640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fac0814be90 0x7fac0814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:55518/0 (socket says 172.21.15.155:55518) 2024-03-25T18:25:20.395 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.392+0000 7fac07fff640 1 -- 172.21.15.155:0/3656013647 learned_addr learned my addr 172.21.15.155:0/3656013647 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:25:20.395 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.395+0000 7fac067fc640 1 -- 172.21.15.155:0/3656013647 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2911224802 0 0) 0x7fac080af940 con 0x7fac0814be90 2024-03-25T18:25:20.396 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.395+0000 7fac067fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:20.396 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.395+0000 7fac067fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:20.396 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.395+0000 7fac067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:20.396 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.395+0000 7fac067fc640 10 monclient(hunting): my global_id is 4221 2024-03-25T18:25:20.396 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.395+0000 7fac067fc640 1 -- 172.21.15.155:0/3656013647 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabf0003520 con 0x7fac0814be90 2024-03-25T18:25:20.397 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.396+0000 7fac067fc640 1 -- 172.21.15.155:0/3656013647 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4114448383 0 0) 0x7fac080b0d50 con 0x7fac080b1730 2024-03-25T18:25:20.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.396+0000 7fac067fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:20.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.396+0000 7fac067fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:20.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.396+0000 7fac067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:20.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.396+0000 7fac067fc640 10 monclient(hunting): my global_id is 4223 2024-03-25T18:25:20.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.396+0000 7fac067fc640 1 -- 172.21.15.155:0/3656013647 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac080af940 con 0x7fac080b1730 2024-03-25T18:25:20.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.396+0000 7fac067fc640 1 -- 172.21.15.155:0/3656013647 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3776359145 0 0) 0x7fac080b0a70 con 0x7fac080b4d60 2024-03-25T18:25:20.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.396+0000 7fac067fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:20.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.396+0000 7fac067fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:20.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.396+0000 7fac067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:20.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.396+0000 7fac067fc640 10 monclient(hunting): my global_id is 4222 2024-03-25T18:25:20.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.396+0000 7fac067fc640 1 -- 172.21.15.155:0/3656013647 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac080b0d50 con 0x7fac080b4d60 2024-03-25T18:25:20.401 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.399+0000 7fac067fc640 1 -- 172.21.15.155:0/3656013647 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1714164279 0 0) 0x7fabf0003520 con 0x7fac0814be90 2024-03-25T18:25:20.401 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.399+0000 7fac067fc640 1 -- 172.21.15.155:0/3656013647 >> v1:172.21.15.155:6790/0 conn(0x7fac080b1730 legacy=0x7fac080b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:20.401 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.399+0000 7fac067fc640 1 -- 172.21.15.155:0/3656013647 >> v1:172.21.15.167:6789/0 conn(0x7fac080b4d60 legacy=0x7fac0813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:20.401 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.399+0000 7fac067fc640 10 monclient: _finish_hunting 0 2024-03-25T18:25:20.401 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.399+0000 7fac067fc640 1 monclient: found mon.noname-a 2024-03-25T18:25:20.401 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.399+0000 7fac067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:20.402 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.399+0000 7fac067fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-25T18:25:20.402 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.399+0000 7fac067fc640 1 -- 172.21.15.155:0/3656013647 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac080b1030 con 0x7fac0814be90 2024-03-25T18:25:20.402 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.399+0000 7fac067fc640 10 monclient: _finish_auth 0 2024-03-25T18:25:20.402 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.399+0000 7fac067fc640 10 monclient: _check_auth_tickets 2024-03-25T18:25:20.402 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.400+0000 7fac067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:20.402 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.400+0000 7fac067fc640 1 -- 172.21.15.155:0/3656013647 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fac00001a10 con 0x7fac0814be90 2024-03-25T18:25:20.402 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.400+0000 7fac067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:20.402 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.400+0000 7fac067fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:20.402 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.400+0000 7fac067fc640 10 monclient: dump: 2024-03-25T18:25:20.402 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:25:20.402 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:25:20.403 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:20.403 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:20.403 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:20.403 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:25:20.403 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:25:20.403 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:25:20.403 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:25:20.403 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:20.403 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.400+0000 7fac0de0f640 5 monclient: authenticate success, global_id 4221 2024-03-25T18:25:20.403 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.400+0000 7fac0de0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:20.404 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.401+0000 7fac067fc640 1 -- 172.21.15.155:0/3656013647 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fac00001c30 con 0x7fac0814be90 2024-03-25T18:25:20.404 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.401+0000 7fac067fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:20.404 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.402+0000 7fac0de0f640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:20.404 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.402+0000 7fac0de0f640 4 set_mon_vals no callback set 2024-03-25T18:25:20.404 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.402+0000 7fac067fc640 1 -- 172.21.15.155:0/3656013647 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fac00001fc0 con 0x7fac0814be90 2024-03-25T18:25:20.405 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.402+0000 7fac067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:20.405 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.402+0000 7fac067fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:20.405 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.402+0000 7fac067fc640 10 monclient: dump: 2024-03-25T18:25:20.405 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:25:20.405 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:25:20.405 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:20.405 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:20.405 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:20.405 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:25:20.405 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:25:20.405 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:25:20.405 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:25:20.405 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:20.405 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.403+0000 7fac0de0f640 10 monclient: shutdown 2024-03-25T18:25:20.405 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.403+0000 7fac0de0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:20.406 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.403+0000 7fac0de0f640 1 -- 172.21.15.155:0/3656013647 >> v1:172.21.15.155:6789/0 conn(0x7fac0814be90 legacy=0x7fac0814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:20.406 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.403+0000 7fac0de0f640 1 -- 172.21.15.155:0/3656013647 shutdown_connections 2024-03-25T18:25:20.406 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.403+0000 7fac0de0f640 1 -- 172.21.15.155:0/3656013647 >> 172.21.15.155:0/3656013647 conn(0x7fac080617a0 msgr2=0x7fac08061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:20.406 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.403+0000 7fac0de0f640 1 -- 172.21.15.155:0/3656013647 shutdown_connections 2024-03-25T18:25:20.406 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.403+0000 7fac0de0f640 1 -- 172.21.15.155:0/3656013647 wait complete. 2024-03-25T18:25:20.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.404+0000 7fac0de0f640 10 monclient: build_initial_monmap 2024-03-25T18:25:20.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.404+0000 7fac0de0f640 10 monclient: monmap: 2024-03-25T18:25:20.408 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:25:20.408 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:20.409 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:25:20.409 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:25:20.409 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:20.409 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:25:20.409 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:25:20.409 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:25:20.409 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:25:20.409 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:20.409 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.404+0000 7fac0de0f640 1 librados: starting msgr at 2024-03-25T18:25:20.409 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.404+0000 7fac0de0f640 1 librados: starting objecter 2024-03-25T18:25:20.409 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.404+0000 7fac0de0f640 1 Processor -- start 2024-03-25T18:25:20.409 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.405+0000 7fac0de0f640 1 -- start start 2024-03-25T18:25:20.409 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.405+0000 7fac0de0f640 1 librados: setting wanted keys 2024-03-25T18:25:20.409 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.405+0000 7fac0de0f640 1 librados: calling monclient init 2024-03-25T18:25:20.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.405+0000 7fac0de0f640 10 monclient: init 2024-03-25T18:25:20.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.405+0000 7fac0de0f640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:20.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.405+0000 7fac0de0f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:25:20.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.405+0000 7fac0de0f640 10 monclient(hunting): picked mon.noname-a con 0x7fac080b1730 addr v1:172.21.15.155:6789/0 2024-03-25T18:25:20.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.405+0000 7fac0de0f640 10 monclient(hunting): picked mon.noname-c con 0x7fac080b4d60 addr v1:172.21.15.155:6790/0 2024-03-25T18:25:20.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.405+0000 7fac0de0f640 10 monclient(hunting): picked mon.noname-b con 0x7fac0814be90 addr v1:172.21.15.167:6789/0 2024-03-25T18:25:20.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.405+0000 7fac0de0f640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac08140860 con 0x7fac080b1730 2024-03-25T18:25:20.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.405+0000 7fac0de0f640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac08137fd0 con 0x7fac0814be90 2024-03-25T18:25:20.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.405+0000 7fac0de0f640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac08140380 con 0x7fac080b4d60 2024-03-25T18:25:20.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.406+0000 7fac0de0f640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:20.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.406+0000 7fac0de0f640 10 monclient(hunting): authenticate will time out at 7566.715777s 2024-03-25T18:25:20.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.406+0000 7fac06ffd640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fac080b4d60 0x7fac08142170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:52924/0 (socket says 172.21.15.155:52924) 2024-03-25T18:25:20.411 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.406+0000 7fac06ffd640 1 -- 172.21.15.155:0/862507222 learned_addr learned my addr 172.21.15.155:0/862507222 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:25:20.411 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.406+0000 7fabeffff640 10 client.?.objecter ms_handle_connect 0x7fac080b4d60 2024-03-25T18:25:20.411 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.406+0000 7fabeffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:20.411 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.406+0000 7fabeffff640 10 client.?.objecter ms_handle_connect 0x7fac0814be90 2024-03-25T18:25:20.411 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.406+0000 7fabeffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:20.411 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.407+0000 7fabeffff640 10 client.?.objecter ms_handle_connect 0x7fac080b1730 2024-03-25T18:25:20.411 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.407+0000 7fabeffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:20.412 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.409+0000 7fabeffff640 1 -- 172.21.15.155:0/862507222 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4046016308 0 0) 0x7fac08140380 con 0x7fac080b4d60 2024-03-25T18:25:20.412 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.409+0000 7fabeffff640 10 monclient(hunting): _negotiate 2024-03-25T18:25:20.412 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.409+0000 7fabeffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:20.412 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.409+0000 7fabeffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:20.412 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.409+0000 7fabeffff640 10 monclient(hunting): my global_id is 4226 2024-03-25T18:25:20.412 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.409+0000 7fabeffff640 1 -- 172.21.15.155:0/862507222 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabf4004eb0 con 0x7fac080b4d60 2024-03-25T18:25:20.413 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.409+0000 7fabeffff640 1 -- 172.21.15.155:0/862507222 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1971323221 0 0) 0x7fac08140860 con 0x7fac080b1730 2024-03-25T18:25:20.413 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.409+0000 7fabeffff640 10 monclient(hunting): _negotiate 2024-03-25T18:25:20.413 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.409+0000 7fabeffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:20.413 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.409+0000 7fabeffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:20.413 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.409+0000 7fabeffff640 10 monclient(hunting): my global_id is 4224 2024-03-25T18:25:20.413 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.409+0000 7fabeffff640 1 -- 172.21.15.155:0/862507222 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabf40051d0 con 0x7fac080b1730 2024-03-25T18:25:20.413 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.409+0000 7fabeffff640 1 -- 172.21.15.155:0/862507222 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3902264617 0 0) 0x7fac08137fd0 con 0x7fac0814be90 2024-03-25T18:25:20.413 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.410+0000 7fabeffff640 10 monclient(hunting): _negotiate 2024-03-25T18:25:20.413 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.410+0000 7fabeffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:20.413 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.410+0000 7fabeffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:20.413 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.410+0000 7fabeffff640 10 monclient(hunting): my global_id is 4225 2024-03-25T18:25:20.413 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.410+0000 7fabeffff640 1 -- 172.21.15.155:0/862507222 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac08140860 con 0x7fac0814be90 2024-03-25T18:25:20.416 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.415+0000 7fabeffff640 1 -- 172.21.15.155:0/862507222 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1995611100 0 0) 0x7fabf4004eb0 con 0x7fac080b4d60 2024-03-25T18:25:20.416 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.415+0000 7fabeffff640 1 -- 172.21.15.155:0/862507222 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac08137fd0 con 0x7fac080b4d60 2024-03-25T18:25:20.416 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.415+0000 7fabeffff640 1 -- 172.21.15.155:0/862507222 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fabfc0019f0 con 0x7fac080b4d60 2024-03-25T18:25:20.416 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.415+0000 7fabeffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:20.416 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.415+0000 7fabeffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:20.416 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.415+0000 7fabeffff640 10 monclient(hunting): dump: 2024-03-25T18:25:20.416 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:25:20.416 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:25:20.416 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:20.417 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:20.417 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:20.417 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:25:20.417 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:25:20.417 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:25:20.417 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:25:20.417 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:20.417 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.415+0000 7fabeffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:20.418 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.416+0000 7fabeffff640 1 -- 172.21.15.155:0/862507222 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 211185192 0 0) 0x7fac08140860 con 0x7fac0814be90 2024-03-25T18:25:20.418 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.416+0000 7fabeffff640 1 -- 172.21.15.155:0/862507222 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fabf4004eb0 con 0x7fac0814be90 2024-03-25T18:25:20.418 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.416+0000 7fabeffff640 1 -- 172.21.15.155:0/862507222 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fac00001da0 con 0x7fac0814be90 2024-03-25T18:25:20.418 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.416+0000 7fabeffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:20.418 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.416+0000 7fabeffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:20.418 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.416+0000 7fabeffff640 10 monclient(hunting): dump: 2024-03-25T18:25:20.419 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:25:20.419 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:25:20.419 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:20.419 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:20.419 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:20.419 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:25:20.419 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:25:20.419 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:25:20.419 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:25:20.419 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:20.419 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.416+0000 7fabeffff640 1 -- 172.21.15.155:0/862507222 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 344462579 0 0) 0x7fabf40051d0 con 0x7fac080b1730 2024-03-25T18:25:20.419 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.416+0000 7fabeffff640 1 -- 172.21.15.155:0/862507222 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac08140860 con 0x7fac080b1730 2024-03-25T18:25:20.419 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.416+0000 7fabeffff640 1 -- 172.21.15.155:0/862507222 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fabf8001870 con 0x7fac080b1730 2024-03-25T18:25:20.419 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.416+0000 7fabeffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:20.419 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.416+0000 7fabeffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:20.419 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.416+0000 7fabeffff640 10 monclient(hunting): dump: 2024-03-25T18:25:20.420 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:25:20.420 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:25:20.420 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:20.420 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:20.420 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:20.420 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:25:20.420 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:25:20.420 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:25:20.420 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:25:20.420 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:20.421 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.420+0000 7fabeffff640 1 -- 172.21.15.155:0/862507222 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1457276366 0 0) 0x7fac08137fd0 con 0x7fac080b4d60 2024-03-25T18:25:20.421 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.420+0000 7fabeffff640 1 -- 172.21.15.155:0/862507222 >> v1:172.21.15.167:6789/0 conn(0x7fac0814be90 legacy=0x7fac08142880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:20.421 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.420+0000 7fabeffff640 1 -- 172.21.15.155:0/862507222 >> v1:172.21.15.155:6789/0 conn(0x7fac080b1730 legacy=0x7fac08134d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:20.421 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.420+0000 7fabeffff640 10 monclient: _finish_hunting 0 2024-03-25T18:25:20.421 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.420+0000 7fabeffff640 1 monclient: found mon.c 2024-03-25T18:25:20.421 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.420+0000 7fabeffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:20.421 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.420+0000 7fabeffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:25:20.421 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.420+0000 7fabeffff640 1 -- 172.21.15.155:0/862507222 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac08140510 con 0x7fac080b4d60 2024-03-25T18:25:20.421 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.420+0000 7fabeffff640 10 monclient: _finish_auth 0 2024-03-25T18:25:20.421 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.420+0000 7fabeffff640 10 monclient: _check_auth_tickets 2024-03-25T18:25:20.421 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.420+0000 7fabeffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:20.421 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.420+0000 7fac0de0f640 5 monclient: authenticate success, global_id 4226 2024-03-25T18:25:20.421 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.420+0000 7fac0de0f640 10 monclient: _renew_subs 2024-03-25T18:25:20.422 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.420+0000 7fac0de0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:25:20.422 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.420+0000 7fac0de0f640 1 -- 172.21.15.155:0/862507222 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac081374a0 con 0x7fac080b4d60 2024-03-25T18:25:20.422 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.420+0000 7fac0de0f640 10 client.4226.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:20.422 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.420+0000 7fac0de0f640 10 monclient: _renew_subs 2024-03-25T18:25:20.422 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.420+0000 7fac0de0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:25:20.422 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.420+0000 7fac0de0f640 1 -- 172.21.15.155:0/862507222 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fac08145000 con 0x7fac080b4d60 2024-03-25T18:25:20.422 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.420+0000 7fac0de0f640 1 librados: init done 2024-03-25T18:25:20.423 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.421+0000 7fac0de0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:20.423 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.421+0000 7fac0de0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:20.423 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.421+0000 7fac0de0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:25:20.423 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.421+0000 7fac0de0f640 1 -- 172.21.15.155:0/862507222 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac080627b0 con 0x7fac080b4d60 2024-03-25T18:25:20.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.670+0000 7fabeffff640 1 -- 172.21.15.155:0/862507222 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fabfc003610 con 0x7fac080b4d60 2024-03-25T18:25:20.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.670+0000 7fabeffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:20.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.670+0000 7fabeffff640 1 -- 172.21.15.155:0/862507222 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fabfc0039a0 con 0x7fac080b4d60 2024-03-25T18:25:20.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.670+0000 7fabeffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:20.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.670+0000 7fabeffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:20.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.670+0000 7fabeffff640 10 monclient: dump: 2024-03-25T18:25:20.671 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:25:20.671 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:25:20.671 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:20.672 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:20.672 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:20.672 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:25:20.672 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:25:20.672 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:25:20.672 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:25:20.672 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:20.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.670+0000 7fac05ffb640 4 set_mon_vals no callback set 2024-03-25T18:25:20.673 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.671+0000 7fabeffff640 1 -- 172.21.15.155:0/862507222 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7fabfc003b80 con 0x7fac080b4d60 2024-03-25T18:25:20.673 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.672+0000 7fac077fe640 10 monclient: get_auth_request con 0x7fabf406e5d0 auth_method 0 2024-03-25T18:25:20.673 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.672+0000 7fabeffff640 10 client.4226.objecter ms_handle_connect 0x7fabf406e5d0 2024-03-25T18:25:20.679 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.678+0000 7fabeffff640 1 -- 172.21.15.155:0/862507222 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7fabfc054830 con 0x7fac080b4d60 2024-03-25T18:25:20.679 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.678+0000 7fabeffff640 10 client.4226.objecter ms_dispatch 0x7fac08000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:20.679 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.678+0000 7fabeffff640 3 client.4226.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:20.679 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.678+0000 7fabeffff640 3 client.4226.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:20.679 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.678+0000 7fabeffff640 20 client.4226.objecter dump_active .. 0 homeless 2024-03-25T18:25:20.679 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.678+0000 7fabeffff640 1 -- 172.21.15.155:0/862507222 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fabfc050170 con 0x7fac080b4d60 2024-03-25T18:25:20.679 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.678+0000 7fabeffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:20.679 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.678+0000 7fabeffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:20.873 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.872+0000 7fac0de0f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:20.873 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.872+0000 7fac0de0f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:20.874 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.872+0000 7fac0de0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:25:20.874 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.872+0000 7fac0de0f640 1 -- 172.21.15.155:0/862507222 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fac08143310 con 0x7fac080b4d60 2024-03-25T18:25:20.889 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.888+0000 7fabeffff640 1 -- 172.21.15.155:0/862507222 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7fabfc052270 con 0x7fac080b4d60 2024-03-25T18:25:20.889 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.888+0000 7fabeffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:20.889 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.889+0000 7fabeffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:20.890 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:25:20.890 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:20.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.891+0000 7fac0de0f640 10 librados: watch_flush enter 2024-03-25T18:25:20.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.891+0000 7fac0de0f640 10 librados: watch_flush exit 2024-03-25T18:25:20.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.891+0000 7fac0de0f640 20 client.4226.objecter shutdown clearing up homeless session... 2024-03-25T18:25:20.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.891+0000 7fac0de0f640 10 client.4226.objecter successfully canceled tick 2024-03-25T18:25:20.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.891+0000 7fac0de0f640 1 -- 172.21.15.155:0/862507222 >> v1:172.21.15.167:6800/2919070503 conn(0x7fabf406e5d0 legacy=0x7fabf4070a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:20.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.891+0000 7fac0de0f640 10 monclient: shutdown 2024-03-25T18:25:20.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.891+0000 7fac0de0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:20.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.891+0000 7fac0de0f640 1 -- 172.21.15.155:0/862507222 >> v1:172.21.15.155:6790/0 conn(0x7fac080b4d60 legacy=0x7fac08142170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:20.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.891+0000 7fac0de0f640 1 -- 172.21.15.155:0/862507222 shutdown_connections 2024-03-25T18:25:20.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.891+0000 7fac0de0f640 1 -- 172.21.15.155:0/862507222 >> 172.21.15.155:0/862507222 conn(0x7fac080617a0 msgr2=0x7fac0813c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:20.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.892+0000 7fac0de0f640 1 -- 172.21.15.155:0/862507222 shutdown_connections 2024-03-25T18:25:20.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.892+0000 7fac0de0f640 1 -- 172.21.15.155:0/862507222 wait complete. 2024-03-25T18:25:20.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:20.892+0000 7fac0de0f640 1 librados: shutdown 2024-03-25T18:25:20.903 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:23.499 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-03-25T18:25:23.494+0000 5dc3480 -1 osd.6 0 log_to_monitors true 2024-03-25T18:25:23.590 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-03-25T18:25:23.586+0000 5dc3480 -1 osd.5 0 log_to_monitors true 2024-03-25T18:25:23.752 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-03-25T18:25:23.737+0000 5dc3480 -1 osd.7 0 log_to_monitors true 2024-03-25T18:25:24.439 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-03-25T18:25:24.435+0000 5dc3480 -1 osd.4 0 log_to_monitors true 2024-03-25T18:25:24.664 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-03-25T18:25:24.658+0000 5dc3480 -1 osd.2 0 log_to_monitors true 2024-03-25T18:25:25.068 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-03-25T18:25:25.063+0000 5dc3480 -1 osd.3 0 log_to_monitors true 2024-03-25T18:25:25.269 INFO:tasks.ceph.osd.1.smithi155.stderr:2024-03-25T18:25:25.265+0000 5dc3480 -1 osd.1 0 log_to_monitors true 2024-03-25T18:25:25.864 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-03-25T18:25:25.859+0000 5dc3480 -1 osd.0 0 log_to_monitors true 2024-03-25T18:25:27.710 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:27.828 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.827+0000 7fa00bbd0640 10 monclient: get_monmap_and_config 2024-03-25T18:25:27.828 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.827+0000 7fa00bbd0640 10 monclient: build_initial_monmap 2024-03-25T18:25:27.828 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.827+0000 7fa00bbd0640 10 monclient: monmap: 2024-03-25T18:25:27.828 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:25:27.828 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:25:27.828 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:25:27.828389+0000 2024-03-25T18:25:27.829 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:25:27.828389+0000 2024-03-25T18:25:27.829 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:27.829 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:25:27.829 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:25:27.829 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:25:27.829 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:25:27.829 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:27.830 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.828+0000 7fa00bbd0640 1 Processor -- start 2024-03-25T18:25:27.830 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.828+0000 7fa00bbd0640 1 -- start start 2024-03-25T18:25:27.830 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.828+0000 7fa00bbd0640 10 monclient: init 2024-03-25T18:25:27.830 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.829+0000 7fa00bbd0640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:27.830 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.829+0000 7fa00bbd0640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:25:27.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.829+0000 7fa00bbd0640 10 monclient(hunting): picked mon.noname-c con 0x7fa0040a98f0 addr v1:172.21.15.167:6789/0 2024-03-25T18:25:27.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.829+0000 7fa00bbd0640 10 monclient(hunting): picked mon.noname-b con 0x7fa00413f710 addr v1:172.21.15.155:6790/0 2024-03-25T18:25:27.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.829+0000 7fa00bbd0640 10 monclient(hunting): picked mon.noname-a con 0x7fa0041430c0 addr v1:172.21.15.155:6789/0 2024-03-25T18:25:27.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.829+0000 7fa00bbd0640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa004142520 con 0x7fa0041430c0 2024-03-25T18:25:27.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.829+0000 7fa00bbd0640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0040b28a0 con 0x7fa0040a98f0 2024-03-25T18:25:27.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.829+0000 7fa00bbd0640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0040b2a30 con 0x7fa00413f710 2024-03-25T18:25:27.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.829+0000 7fa00bbd0640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:27.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.829+0000 7fa00bbd0640 10 monclient(hunting): authenticate will time out at 7574.138989s 2024-03-25T18:25:27.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.831+0000 7fa00996c640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7fa0040a98f0 0x7fa0040a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:45666/0 (socket says 172.21.15.155:45666) 2024-03-25T18:25:27.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.832+0000 7fa00996c640 1 -- 172.21.15.155:0/2231761026 learned_addr learned my addr 172.21.15.155:0/2231761026 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:25:27.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.846+0000 7fa00896a640 1 -- 172.21.15.155:0/2231761026 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 487313457 0 0) 0x7fa0040b28a0 con 0x7fa0040a98f0 2024-03-25T18:25:27.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.846+0000 7fa00896a640 10 monclient(hunting): _negotiate 2024-03-25T18:25:27.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.846+0000 7fa00896a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:27.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.846+0000 7fa00896a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:27.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.846+0000 7fa00896a640 10 monclient(hunting): my global_id is 4252 2024-03-25T18:25:27.848 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.846+0000 7fa00896a640 1 -- 172.21.15.155:0/2231761026 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ff0003520 con 0x7fa0040a98f0 2024-03-25T18:25:27.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.854+0000 7fa00896a640 1 -- 172.21.15.155:0/2231761026 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1735954264 0 0) 0x7fa004142520 con 0x7fa0041430c0 2024-03-25T18:25:27.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.854+0000 7fa00896a640 10 monclient(hunting): _negotiate 2024-03-25T18:25:27.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.854+0000 7fa00896a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:27.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.854+0000 7fa00896a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:27.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.854+0000 7fa00896a640 10 monclient(hunting): my global_id is 4251 2024-03-25T18:25:27.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.854+0000 7fa00896a640 1 -- 172.21.15.155:0/2231761026 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0040b28a0 con 0x7fa0041430c0 2024-03-25T18:25:27.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.857+0000 7fa00896a640 1 -- 172.21.15.155:0/2231761026 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2930711524 0 0) 0x7f9ff0003520 con 0x7fa0040a98f0 2024-03-25T18:25:27.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.857+0000 7fa00896a640 1 -- 172.21.15.155:0/2231761026 >> v1:172.21.15.155:6790/0 conn(0x7fa00413f710 legacy=0x7fa0041429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:27.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.857+0000 7fa00896a640 1 -- 172.21.15.155:0/2231761026 >> v1:172.21.15.155:6789/0 conn(0x7fa0041430c0 legacy=0x7fa00414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:27.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.857+0000 7fa00896a640 10 monclient: _finish_hunting 0 2024-03-25T18:25:27.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.857+0000 7fa00896a640 1 monclient: found mon.noname-c 2024-03-25T18:25:27.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.857+0000 7fa00896a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:27.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.857+0000 7fa00896a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.167:6789/0 2024-03-25T18:25:27.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.857+0000 7fa00896a640 1 -- 172.21.15.155:0/2231761026 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0040b44c0 con 0x7fa0040a98f0 2024-03-25T18:25:27.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.857+0000 7fa00896a640 10 monclient: _finish_auth 0 2024-03-25T18:25:27.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.857+0000 7fa00896a640 10 monclient: _check_auth_tickets 2024-03-25T18:25:27.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.857+0000 7fa00896a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:27.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.857+0000 7fa00896a640 1 -- 172.21.15.155:0/2231761026 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fa000001690 con 0x7fa0040a98f0 2024-03-25T18:25:27.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.857+0000 7fa00bbd0640 5 monclient: authenticate success, global_id 4252 2024-03-25T18:25:27.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.857+0000 7fa00bbd0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:27.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.857+0000 7fa00896a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:27.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.857+0000 7fa00896a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:27.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.857+0000 7fa00896a640 10 monclient: dump: 2024-03-25T18:25:27.859 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:25:27.859 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:25:27.859 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:27.860 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:27.860 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:27.860 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:25:27.860 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:25:27.860 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:25:27.860 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:25:27.860 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:27.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.857+0000 7fa00bbd0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:27.876 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.875+0000 7fa00896a640 1 -- 172.21.15.155:0/2231761026 <== mon.1 v1:172.21.15.167:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa0000017f0 con 0x7fa0040a98f0 2024-03-25T18:25:27.876 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.875+0000 7fa00896a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:27.876 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.875+0000 7fa00896a640 1 -- 172.21.15.155:0/2231761026 <== mon.1 v1:172.21.15.167:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fa000001c00 con 0x7fa0040a98f0 2024-03-25T18:25:27.876 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.875+0000 7fa00896a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:27.876 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.875+0000 7fa00896a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:27.876 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.875+0000 7fa00896a640 10 monclient: dump: 2024-03-25T18:25:27.876 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:25:27.876 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:25:27.876 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:27.877 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:27.877 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:27.877 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:25:27.877 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:25:27.877 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:25:27.877 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:25:27.877 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:27.877 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.875+0000 7fa00bbd0640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:27.877 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.875+0000 7fa00bbd0640 4 set_mon_vals no callback set 2024-03-25T18:25:27.879 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.876+0000 7fa00bbd0640 10 monclient: shutdown 2024-03-25T18:25:27.879 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.876+0000 7fa00bbd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:27.879 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.876+0000 7fa00bbd0640 1 -- 172.21.15.155:0/2231761026 >> v1:172.21.15.167:6789/0 conn(0x7fa0040a98f0 legacy=0x7fa0040a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:27.879 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.876+0000 7fa00bbd0640 1 -- 172.21.15.155:0/2231761026 shutdown_connections 2024-03-25T18:25:27.879 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.876+0000 7fa00bbd0640 1 -- 172.21.15.155:0/2231761026 >> 172.21.15.155:0/2231761026 conn(0x7fa0040b7ea0 msgr2=0x7fa0040b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:27.879 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.876+0000 7fa00bbd0640 1 -- 172.21.15.155:0/2231761026 shutdown_connections 2024-03-25T18:25:27.879 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.876+0000 7fa00bbd0640 1 -- 172.21.15.155:0/2231761026 wait complete. 2024-03-25T18:25:27.879 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.876+0000 7fa00bbd0640 10 monclient: build_initial_monmap 2024-03-25T18:25:27.879 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.876+0000 7fa00bbd0640 10 monclient: monmap: 2024-03-25T18:25:27.879 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:25:27.879 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:27.880 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:25:27.880 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:25:27.880 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:27.880 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:25:27.880 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:25:27.880 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:25:27.880 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:25:27.880 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:27.880 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.877+0000 7fa00bbd0640 1 librados: starting msgr at 2024-03-25T18:25:27.880 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.877+0000 7fa00bbd0640 1 librados: starting objecter 2024-03-25T18:25:27.880 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.877+0000 7fa00bbd0640 1 Processor -- start 2024-03-25T18:25:27.880 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.877+0000 7fa00bbd0640 1 -- start start 2024-03-25T18:25:27.880 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.877+0000 7fa00bbd0640 1 librados: setting wanted keys 2024-03-25T18:25:27.880 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.877+0000 7fa00bbd0640 1 librados: calling monclient init 2024-03-25T18:25:27.881 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.877+0000 7fa00bbd0640 10 monclient: init 2024-03-25T18:25:27.881 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.877+0000 7fa00bbd0640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:27.881 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.877+0000 7fa00bbd0640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:25:27.881 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.877+0000 7fa00bbd0640 10 monclient(hunting): picked mon.noname-b con 0x7fa0040a98f0 addr v1:172.21.15.167:6789/0 2024-03-25T18:25:27.881 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.877+0000 7fa00bbd0640 10 monclient(hunting): picked mon.noname-a con 0x7fa00413f710 addr v1:172.21.15.155:6789/0 2024-03-25T18:25:27.881 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.877+0000 7fa00bbd0640 10 monclient(hunting): picked mon.noname-c con 0x7fa0041430c0 addr v1:172.21.15.155:6790/0 2024-03-25T18:25:27.881 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.877+0000 7fa00bbd0640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa004138250 con 0x7fa00413f710 2024-03-25T18:25:27.881 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.878+0000 7fa00bbd0640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa004137d50 con 0x7fa0040a98f0 2024-03-25T18:25:27.881 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.878+0000 7fa00bbd0640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa004137ee0 con 0x7fa0041430c0 2024-03-25T18:25:27.881 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.878+0000 7fa00bbd0640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:27.881 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.878+0000 7fa00bbd0640 10 monclient(hunting): authenticate will time out at 7574.187741s 2024-03-25T18:25:27.889 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.888+0000 7fa00996c640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7fa0040a98f0 0x7fa004137640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:45726/0 (socket says 172.21.15.155:45726) 2024-03-25T18:25:27.889 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.888+0000 7fa00996c640 1 -- 172.21.15.155:0/3916603596 learned_addr learned my addr 172.21.15.155:0/3916603596 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:25:27.890 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.890+0000 7f9ffa7fc640 10 client.?.objecter ms_handle_connect 0x7fa0040a98f0 2024-03-25T18:25:27.890 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.890+0000 7f9ffa7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:27.892 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.891+0000 7f9ffa7fc640 10 client.?.objecter ms_handle_connect 0x7fa00413f710 2024-03-25T18:25:27.892 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.891+0000 7f9ffa7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:27.898 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.897+0000 7f9ffa7fc640 10 client.?.objecter ms_handle_connect 0x7fa0041430c0 2024-03-25T18:25:27.898 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.897+0000 7f9ffa7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:27.902 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.901+0000 7f9ffa7fc640 1 -- 172.21.15.155:0/3916603596 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1734145708 0 0) 0x7fa004138250 con 0x7fa00413f710 2024-03-25T18:25:27.902 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.901+0000 7f9ffa7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:27.902 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.901+0000 7f9ffa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:27.902 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.901+0000 7f9ffa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:27.902 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.901+0000 7f9ffa7fc640 10 monclient(hunting): my global_id is 4254 2024-03-25T18:25:27.902 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.901+0000 7f9ffa7fc640 1 -- 172.21.15.155:0/3916603596 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fe8004eb0 con 0x7fa00413f710 2024-03-25T18:25:27.906 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.905+0000 7f9ffa7fc640 1 -- 172.21.15.155:0/3916603596 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2562470242 0 0) 0x7fa004137d50 con 0x7fa0040a98f0 2024-03-25T18:25:27.906 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.905+0000 7f9ffa7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:27.906 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.905+0000 7f9ffa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:27.906 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.905+0000 7f9ffa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:27.906 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.905+0000 7f9ffa7fc640 10 monclient(hunting): my global_id is 4255 2024-03-25T18:25:27.906 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.905+0000 7f9ffa7fc640 1 -- 172.21.15.155:0/3916603596 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa004138250 con 0x7fa0040a98f0 2024-03-25T18:25:27.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.917+0000 7f9ffa7fc640 1 -- 172.21.15.155:0/3916603596 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 735890339 0 0) 0x7fa004137ee0 con 0x7fa0041430c0 2024-03-25T18:25:27.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.917+0000 7f9ffa7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:27.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.917+0000 7f9ffa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:27.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.917+0000 7f9ffa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:27.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.917+0000 7f9ffa7fc640 10 monclient(hunting): my global_id is 4256 2024-03-25T18:25:27.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.917+0000 7f9ffa7fc640 1 -- 172.21.15.155:0/3916603596 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa004137d50 con 0x7fa0041430c0 2024-03-25T18:25:27.923 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.922+0000 7f9ffa7fc640 1 -- 172.21.15.155:0/3916603596 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1564861636 0 0) 0x7f9fe8004eb0 con 0x7fa00413f710 2024-03-25T18:25:27.923 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.922+0000 7f9ffa7fc640 1 -- 172.21.15.155:0/3916603596 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa004137ee0 con 0x7fa00413f710 2024-03-25T18:25:27.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.922+0000 7f9ffa7fc640 1 -- 172.21.15.155:0/3916603596 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f9ff40018c0 con 0x7fa00413f710 2024-03-25T18:25:27.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.922+0000 7f9ffa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:27.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.922+0000 7f9ffa7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:27.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.922+0000 7f9ffa7fc640 10 monclient(hunting): dump: 2024-03-25T18:25:27.924 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:25:27.924 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:25:27.924 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:27.924 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:27.924 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:27.924 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:25:27.924 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:25:27.924 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:25:27.924 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:25:27.924 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:27.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.922+0000 7f9ffa7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:27.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.923+0000 7f9ffa7fc640 1 -- 172.21.15.155:0/3916603596 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2694777189 0 0) 0x7fa004138250 con 0x7fa0040a98f0 2024-03-25T18:25:27.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.923+0000 7f9ffa7fc640 1 -- 172.21.15.155:0/3916603596 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9fe8004eb0 con 0x7fa0040a98f0 2024-03-25T18:25:27.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.923+0000 7f9ffa7fc640 1 -- 172.21.15.155:0/3916603596 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fa000001bc0 con 0x7fa0040a98f0 2024-03-25T18:25:27.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.923+0000 7f9ffa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:27.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.923+0000 7f9ffa7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:27.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.923+0000 7f9ffa7fc640 10 monclient(hunting): dump: 2024-03-25T18:25:27.925 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:25:27.925 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:25:27.925 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:27.925 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:27.925 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:27.925 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:25:27.925 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:25:27.925 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:25:27.926 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:25:27.926 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:27.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.929+0000 7f9ffa7fc640 1 -- 172.21.15.155:0/3916603596 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2699675808 0 0) 0x7fa004137d50 con 0x7fa0041430c0 2024-03-25T18:25:27.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.929+0000 7f9ffa7fc640 1 -- 172.21.15.155:0/3916603596 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa004138250 con 0x7fa0041430c0 2024-03-25T18:25:27.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.929+0000 7f9ffa7fc640 1 -- 172.21.15.155:0/3916603596 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f9ffc001bd0 con 0x7fa0041430c0 2024-03-25T18:25:27.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.929+0000 7f9ffa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:27.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.929+0000 7f9ffa7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:27.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.929+0000 7f9ffa7fc640 10 monclient(hunting): dump: 2024-03-25T18:25:27.930 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:25:27.930 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:25:27.930 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:27.930 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:27.930 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:27.930 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:25:27.931 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:25:27.931 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:25:27.931 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:25:27.931 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:27.937 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.935+0000 7f9ffa7fc640 1 -- 172.21.15.155:0/3916603596 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1945596033 0 0) 0x7fa004137ee0 con 0x7fa00413f710 2024-03-25T18:25:27.937 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.935+0000 7f9ffa7fc640 1 -- 172.21.15.155:0/3916603596 >> v1:172.21.15.155:6790/0 conn(0x7fa0041430c0 legacy=0x7fa0041d7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:27.937 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.935+0000 7f9ffa7fc640 1 -- 172.21.15.155:0/3916603596 >> v1:172.21.15.167:6789/0 conn(0x7fa0040a98f0 legacy=0x7fa004137640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:27.937 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.935+0000 7f9ffa7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:25:27.937 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.935+0000 7f9ffa7fc640 1 monclient: found mon.a 2024-03-25T18:25:27.937 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.935+0000 7f9ffa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:27.937 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.935+0000 7f9ffa7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:25:27.937 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.935+0000 7f9ffa7fc640 1 -- 172.21.15.155:0/3916603596 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0040b7730 con 0x7fa00413f710 2024-03-25T18:25:27.937 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.935+0000 7f9ffa7fc640 10 monclient: _finish_auth 0 2024-03-25T18:25:27.938 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.935+0000 7f9ffa7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:25:27.938 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.935+0000 7f9ffa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:27.938 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.935+0000 7fa00bbd0640 5 monclient: authenticate success, global_id 4254 2024-03-25T18:25:27.938 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.936+0000 7fa00bbd0640 10 monclient: _renew_subs 2024-03-25T18:25:27.938 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.936+0000 7fa00bbd0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:25:27.938 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.936+0000 7fa00bbd0640 1 -- 172.21.15.155:0/3916603596 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0040b7920 con 0x7fa00413f710 2024-03-25T18:25:27.938 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.936+0000 7fa00bbd0640 10 client.4254.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:27.938 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.936+0000 7fa00bbd0640 10 monclient: _renew_subs 2024-03-25T18:25:27.938 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.936+0000 7fa00bbd0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:25:27.938 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.936+0000 7fa00bbd0640 1 -- 172.21.15.155:0/3916603596 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0041e33d0 con 0x7fa00413f710 2024-03-25T18:25:27.938 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.936+0000 7fa00bbd0640 1 librados: init done 2024-03-25T18:25:27.939 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.937+0000 7fa00bbd0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:27.939 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.937+0000 7fa00bbd0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:27.939 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.937+0000 7fa00bbd0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:25:27.939 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:27.937+0000 7fa00bbd0640 1 -- 172.21.15.155:0/3916603596 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0040627b0 con 0x7fa00413f710 2024-03-25T18:25:28.236 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:28.235+0000 7f9ffa7fc640 1 -- 172.21.15.155:0/3916603596 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9ff40035f0 con 0x7fa00413f710 2024-03-25T18:25:28.236 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:28.235+0000 7f9ffa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:28.236 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:28.235+0000 7f9ffa7fc640 1 -- 172.21.15.155:0/3916603596 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f9ff4003900 con 0x7fa00413f710 2024-03-25T18:25:28.236 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:28.235+0000 7f9ffa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:28.236 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:28.235+0000 7f9ffa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:28.236 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:28.235+0000 7f9ffa7fc640 10 monclient: dump: 2024-03-25T18:25:28.236 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:25:28.236 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:25:28.236 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:28.237 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:28.237 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:28.237 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:25:28.237 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:25:28.237 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:25:28.237 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:25:28.237 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:28.237 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:28.235+0000 7f9ffbfff640 4 set_mon_vals no callback set 2024-03-25T18:25:28.238 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:28.236+0000 7f9ffa7fc640 1 -- 172.21.15.155:0/3916603596 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7f9ff40198d0 con 0x7fa00413f710 2024-03-25T18:25:28.238 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:28.236+0000 7fa00996c640 10 monclient: get_auth_request con 0x7f9fe806e0d0 auth_method 0 2024-03-25T18:25:28.238 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:28.236+0000 7f9ffa7fc640 10 client.4254.objecter ms_handle_connect 0x7f9fe806e0d0 2024-03-25T18:25:28.242 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:28.241+0000 7f9ffa7fc640 1 -- 172.21.15.155:0/3916603596 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7f9ff4053610 con 0x7fa00413f710 2024-03-25T18:25:28.243 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:28.241+0000 7f9ffa7fc640 10 client.4254.objecter ms_dispatch 0x7fa004000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:28.243 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:28.241+0000 7f9ffa7fc640 3 client.4254.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:28.243 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:28.241+0000 7f9ffa7fc640 3 client.4254.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:28.243 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:28.242+0000 7f9ffa7fc640 20 client.4254.objecter dump_active .. 0 homeless 2024-03-25T18:25:28.243 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:28.242+0000 7f9ffa7fc640 1 -- 172.21.15.155:0/3916603596 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9ff404ef50 con 0x7fa00413f710 2024-03-25T18:25:28.243 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:28.242+0000 7f9ffa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:28.243 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:28.242+0000 7f9ffa7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:28.497 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:28.496+0000 7fa00bbd0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:28.497 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:28.496+0000 7fa00bbd0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:28.497 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:28.496+0000 7fa00bbd0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:25:28.497 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:28.496+0000 7fa00bbd0640 1 -- 172.21.15.155:0/3916603596 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa00413df50 con 0x7fa00413f710 2024-03-25T18:25:28.569 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:28.568+0000 7f9ffa7fc640 1 -- 172.21.15.155:0/3916603596 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7f9ff4051050 con 0x7fa00413f710 2024-03-25T18:25:28.570 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:28.568+0000 7f9ffa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:28.570 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:28.568+0000 7f9ffa7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:28.570 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:25:28.570 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:28.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:28.571+0000 7fa00bbd0640 10 librados: watch_flush enter 2024-03-25T18:25:28.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:28.571+0000 7fa00bbd0640 10 librados: watch_flush exit 2024-03-25T18:25:28.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:28.571+0000 7fa00bbd0640 20 client.4254.objecter shutdown clearing up homeless session... 2024-03-25T18:25:28.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:28.571+0000 7fa00bbd0640 10 client.4254.objecter successfully canceled tick 2024-03-25T18:25:28.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:28.571+0000 7fa00bbd0640 1 -- 172.21.15.155:0/3916603596 >> v1:172.21.15.167:6800/2919070503 conn(0x7f9fe806e0d0 legacy=0x7f9fe8070590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:28.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:28.571+0000 7fa00bbd0640 10 monclient: shutdown 2024-03-25T18:25:28.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:28.571+0000 7fa00bbd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:28.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:28.571+0000 7fa00bbd0640 1 -- 172.21.15.155:0/3916603596 >> v1:172.21.15.155:6789/0 conn(0x7fa00413f710 legacy=0x7fa0041d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:28.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:28.571+0000 7fa00bbd0640 1 -- 172.21.15.155:0/3916603596 shutdown_connections 2024-03-25T18:25:28.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:28.571+0000 7fa00bbd0640 1 -- 172.21.15.155:0/3916603596 >> 172.21.15.155:0/3916603596 conn(0x7fa0040b7ea0 msgr2=0x7fa004074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:28.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:28.572+0000 7fa00bbd0640 1 -- 172.21.15.155:0/3916603596 shutdown_connections 2024-03-25T18:25:28.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:28.572+0000 7fa00bbd0640 1 -- 172.21.15.155:0/3916603596 wait complete. 2024-03-25T18:25:28.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:28.572+0000 7fa00bbd0640 1 librados: shutdown 2024-03-25T18:25:28.583 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:35.391 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:35.502 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.500+0000 7f347aca8640 10 monclient: get_monmap_and_config 2024-03-25T18:25:35.502 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.500+0000 7f347aca8640 10 monclient: build_initial_monmap 2024-03-25T18:25:35.502 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.500+0000 7f347aca8640 10 monclient: monmap: 2024-03-25T18:25:35.502 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:25:35.502 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:25:35.502 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:25:35.502285+0000 2024-03-25T18:25:35.502 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:25:35.502285+0000 2024-03-25T18:25:35.502 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:35.503 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:25:35.503 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:25:35.503 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:25:35.503 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:25:35.503 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:35.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.501+0000 7f347aca8640 1 Processor -- start 2024-03-25T18:25:35.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.501+0000 7f347aca8640 1 -- start start 2024-03-25T18:25:35.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.501+0000 7f347aca8640 10 monclient: init 2024-03-25T18:25:35.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.502+0000 7f347aca8640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:35.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.502+0000 7f347aca8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:25:35.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.502+0000 7f347aca8640 10 monclient(hunting): picked mon.noname-b con 0x7f34740b4d60 addr v1:172.21.15.155:6790/0 2024-03-25T18:25:35.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.502+0000 7f347aca8640 10 monclient(hunting): picked mon.noname-c con 0x7f34740b1730 addr v1:172.21.15.167:6789/0 2024-03-25T18:25:35.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.502+0000 7f347aca8640 10 monclient(hunting): picked mon.noname-a con 0x7f347414be90 addr v1:172.21.15.155:6789/0 2024-03-25T18:25:35.505 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.502+0000 7f347aca8640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34740af940 con 0x7f347414be90 2024-03-25T18:25:35.505 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.502+0000 7f347aca8640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34740b0a70 con 0x7f34740b1730 2024-03-25T18:25:35.505 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.502+0000 7f347aca8640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34740b0d50 con 0x7f34740b4d60 2024-03-25T18:25:35.505 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.502+0000 7f347aca8640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:35.505 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.502+0000 7f347aca8640 10 monclient(hunting): authenticate will time out at 7581.812867s 2024-03-25T18:25:35.516 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.514+0000 7f3473fff640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f34740b1730 0x7f34740b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:50014/0 (socket says 172.21.15.155:50014) 2024-03-25T18:25:35.516 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.514+0000 7f3473fff640 1 -- 172.21.15.155:0/401989786 learned_addr learned my addr 172.21.15.155:0/401989786 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:25:35.531 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.529+0000 7f34737fe640 1 -- 172.21.15.155:0/401989786 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1763486806 0 0) 0x7f34740b0a70 con 0x7f34740b1730 2024-03-25T18:25:35.531 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.529+0000 7f34737fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:35.531 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.529+0000 7f34737fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:35.531 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.529+0000 7f34737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:35.531 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.529+0000 7f34737fe640 10 monclient(hunting): my global_id is 4258 2024-03-25T18:25:35.531 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.529+0000 7f34737fe640 1 -- 172.21.15.155:0/401989786 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3464003520 con 0x7f34740b1730 2024-03-25T18:25:35.535 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.534+0000 7f34737fe640 1 -- 172.21.15.155:0/401989786 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2754374430 0 0) 0x7f34740b0d50 con 0x7f34740b4d60 2024-03-25T18:25:35.536 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.534+0000 7f34737fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:35.536 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.534+0000 7f34737fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:35.536 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.534+0000 7f34737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:35.536 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.534+0000 7f34737fe640 10 monclient(hunting): my global_id is 4259 2024-03-25T18:25:35.536 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.534+0000 7f34737fe640 1 -- 172.21.15.155:0/401989786 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34740b0a70 con 0x7f34740b4d60 2024-03-25T18:25:35.537 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.535+0000 7f34737fe640 1 -- 172.21.15.155:0/401989786 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 637768020 0 0) 0x7f34740af940 con 0x7f347414be90 2024-03-25T18:25:35.537 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.535+0000 7f34737fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:35.537 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.535+0000 7f34737fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:35.537 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.535+0000 7f34737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:35.537 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.535+0000 7f34737fe640 10 monclient(hunting): my global_id is 4257 2024-03-25T18:25:35.537 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.535+0000 7f34737fe640 1 -- 172.21.15.155:0/401989786 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34740b0d50 con 0x7f347414be90 2024-03-25T18:25:35.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.543+0000 7f34737fe640 1 -- 172.21.15.155:0/401989786 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2039490460 0 0) 0x7f34740b0a70 con 0x7f34740b4d60 2024-03-25T18:25:35.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.543+0000 7f34737fe640 1 -- 172.21.15.155:0/401989786 >> v1:172.21.15.167:6789/0 conn(0x7f34740b1730 legacy=0x7f34740b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:35.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.543+0000 7f34737fe640 1 -- 172.21.15.155:0/401989786 >> v1:172.21.15.155:6789/0 conn(0x7f347414be90 legacy=0x7f347414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:35.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.543+0000 7f34737fe640 10 monclient: _finish_hunting 0 2024-03-25T18:25:35.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.543+0000 7f34737fe640 1 monclient: found mon.noname-b 2024-03-25T18:25:35.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.543+0000 7f34737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:35.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.543+0000 7f34737fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-25T18:25:35.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.543+0000 7f34737fe640 1 -- 172.21.15.155:0/401989786 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34740b1030 con 0x7f34740b4d60 2024-03-25T18:25:35.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.543+0000 7f34737fe640 10 monclient: _finish_auth 0 2024-03-25T18:25:35.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.543+0000 7f34737fe640 10 monclient: _check_auth_tickets 2024-03-25T18:25:35.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.543+0000 7f34737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:35.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.543+0000 7f347aca8640 5 monclient: authenticate success, global_id 4259 2024-03-25T18:25:35.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.543+0000 7f34737fe640 1 -- 172.21.15.155:0/401989786 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f3468001670 con 0x7f34740b4d60 2024-03-25T18:25:35.547 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.543+0000 7f347aca8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:35.547 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.543+0000 7f34737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:35.547 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.543+0000 7f34737fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:25:35.547 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.543+0000 7f34737fe640 10 monclient: dump: 2024-03-25T18:25:35.547 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:25:35.547 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:25:35.547 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:35.547 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:35.547 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:35.547 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:25:35.547 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:25:35.547 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:25:35.547 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:25:35.548 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:35.548 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.544+0000 7f347aca8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:35.569 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.568+0000 7f34737fe640 1 -- 172.21.15.155:0/401989786 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3468001890 con 0x7f34740b4d60 2024-03-25T18:25:35.570 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.568+0000 7f34737fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:35.570 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.568+0000 7f34737fe640 1 -- 172.21.15.155:0/401989786 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f3468001c20 con 0x7f34740b4d60 2024-03-25T18:25:35.570 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.568+0000 7f347aca8640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:35.570 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.568+0000 7f347aca8640 4 set_mon_vals no callback set 2024-03-25T18:25:35.570 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.568+0000 7f34737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:35.570 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.568+0000 7f34737fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:35.570 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.568+0000 7f34737fe640 10 monclient: dump: 2024-03-25T18:25:35.570 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:25:35.570 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:25:35.570 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:35.570 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:35.570 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:35.571 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:25:35.571 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:25:35.571 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:25:35.571 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:25:35.571 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:35.572 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.568+0000 7f347aca8640 10 monclient: shutdown 2024-03-25T18:25:35.572 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.568+0000 7f347aca8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:35.572 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.568+0000 7f347aca8640 1 -- 172.21.15.155:0/401989786 >> v1:172.21.15.155:6790/0 conn(0x7f34740b4d60 legacy=0x7f347413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:35.572 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.569+0000 7f347aca8640 1 -- 172.21.15.155:0/401989786 shutdown_connections 2024-03-25T18:25:35.572 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.569+0000 7f347aca8640 1 -- 172.21.15.155:0/401989786 >> 172.21.15.155:0/401989786 conn(0x7f34740617a0 msgr2=0x7f3474061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:35.572 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.569+0000 7f347aca8640 1 -- 172.21.15.155:0/401989786 shutdown_connections 2024-03-25T18:25:35.572 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.569+0000 7f347aca8640 1 -- 172.21.15.155:0/401989786 wait complete. 2024-03-25T18:25:35.572 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.570+0000 7f347aca8640 10 monclient: build_initial_monmap 2024-03-25T18:25:35.572 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.570+0000 7f347aca8640 10 monclient: monmap: 2024-03-25T18:25:35.572 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:25:35.572 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:35.572 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:25:35.572 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:25:35.573 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:35.573 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:25:35.573 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:25:35.573 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:25:35.573 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:25:35.573 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:35.574 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.570+0000 7f347aca8640 1 librados: starting msgr at 2024-03-25T18:25:35.575 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.570+0000 7f347aca8640 1 librados: starting objecter 2024-03-25T18:25:35.575 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.571+0000 7f347aca8640 1 Processor -- start 2024-03-25T18:25:35.575 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.571+0000 7f347aca8640 1 -- start start 2024-03-25T18:25:35.575 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.571+0000 7f347aca8640 1 librados: setting wanted keys 2024-03-25T18:25:35.575 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.571+0000 7f347aca8640 1 librados: calling monclient init 2024-03-25T18:25:35.575 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.571+0000 7f347aca8640 10 monclient: init 2024-03-25T18:25:35.575 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.572+0000 7f347aca8640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:35.575 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.572+0000 7f347aca8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:25:35.575 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.572+0000 7f347aca8640 10 monclient(hunting): picked mon.noname-a con 0x7f34740b1730 addr v1:172.21.15.155:6789/0 2024-03-25T18:25:35.575 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.572+0000 7f347aca8640 10 monclient(hunting): picked mon.noname-b con 0x7f34740b4d60 addr v1:172.21.15.167:6789/0 2024-03-25T18:25:35.575 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.572+0000 7f347aca8640 10 monclient(hunting): picked mon.noname-c con 0x7f347414be90 addr v1:172.21.15.155:6790/0 2024-03-25T18:25:35.575 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.572+0000 7f347aca8640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34741d74b0 con 0x7f34740b1730 2024-03-25T18:25:35.575 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.572+0000 7f347aca8640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34741381e0 con 0x7f34740b4d60 2024-03-25T18:25:35.575 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.572+0000 7f347aca8640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34741384c0 con 0x7f347414be90 2024-03-25T18:25:35.576 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.572+0000 7f347aca8640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:35.576 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.572+0000 7f347aca8640 10 monclient(hunting): authenticate will time out at 7581.883054s 2024-03-25T18:25:35.580 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.578+0000 7f3473fff640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f34740b4d60 0x7f34741367b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:50054/0 (socket says 172.21.15.155:50054) 2024-03-25T18:25:35.580 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.578+0000 7f3473fff640 1 -- 172.21.15.155:0/30144527 learned_addr learned my addr 172.21.15.155:0/30144527 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:25:35.583 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.581+0000 7f34717fa640 10 client.?.objecter ms_handle_connect 0x7f34740b4d60 2024-03-25T18:25:35.583 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.581+0000 7f34717fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:35.589 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.587+0000 7f34717fa640 10 client.?.objecter ms_handle_connect 0x7f34740b1730 2024-03-25T18:25:35.589 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.587+0000 7f34717fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:35.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.591+0000 7f34717fa640 1 -- 172.21.15.155:0/30144527 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3777430976 0 0) 0x7f34741381e0 con 0x7f34740b4d60 2024-03-25T18:25:35.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.591+0000 7f34717fa640 10 monclient(hunting): _negotiate 2024-03-25T18:25:35.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.591+0000 7f34717fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:35.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.591+0000 7f34717fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:35.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.591+0000 7f34717fa640 10 monclient(hunting): my global_id is 4261 2024-03-25T18:25:35.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.591+0000 7f34717fa640 1 -- 172.21.15.155:0/30144527 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3458004eb0 con 0x7f34740b4d60 2024-03-25T18:25:35.595 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.593+0000 7f34717fa640 10 client.?.objecter ms_handle_connect 0x7f347414be90 2024-03-25T18:25:35.595 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.593+0000 7f34717fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:35.609 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.607+0000 7f34717fa640 1 -- 172.21.15.155:0/30144527 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3739230838 0 0) 0x7f3458004eb0 con 0x7f34740b4d60 2024-03-25T18:25:35.609 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.607+0000 7f34717fa640 1 -- 172.21.15.155:0/30144527 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34741381e0 con 0x7f34740b4d60 2024-03-25T18:25:35.609 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.607+0000 7f34717fa640 1 -- 172.21.15.155:0/30144527 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f3460001950 con 0x7f34740b4d60 2024-03-25T18:25:35.609 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.607+0000 7f34717fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:35.609 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.607+0000 7f34717fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:25:35.609 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.607+0000 7f34717fa640 10 monclient(hunting): dump: 2024-03-25T18:25:35.609 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:25:35.609 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:25:35.609 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:35.609 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:35.609 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:35.610 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:25:35.610 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:25:35.610 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:25:35.610 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:25:35.610 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:35.610 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.607+0000 7f34717fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:35.610 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.608+0000 7f34717fa640 1 -- 172.21.15.155:0/30144527 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1878912898 0 0) 0x7f34741384c0 con 0x7f347414be90 2024-03-25T18:25:35.611 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.608+0000 7f34717fa640 10 monclient(hunting): _negotiate 2024-03-25T18:25:35.611 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.608+0000 7f34717fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:35.611 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.608+0000 7f34717fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:35.611 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.608+0000 7f34717fa640 10 monclient(hunting): my global_id is 4262 2024-03-25T18:25:35.611 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.608+0000 7f34717fa640 1 -- 172.21.15.155:0/30144527 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3458004eb0 con 0x7f347414be90 2024-03-25T18:25:35.617 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.615+0000 7f34717fa640 1 -- 172.21.15.155:0/30144527 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3833151098 0 0) 0x7f34741d74b0 con 0x7f34740b1730 2024-03-25T18:25:35.617 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.615+0000 7f34717fa640 10 monclient(hunting): _negotiate 2024-03-25T18:25:35.617 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.615+0000 7f34717fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:35.617 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.615+0000 7f34717fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:35.617 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.615+0000 7f34717fa640 10 monclient(hunting): my global_id is 4260 2024-03-25T18:25:35.617 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.615+0000 7f34717fa640 1 -- 172.21.15.155:0/30144527 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34741384c0 con 0x7f34740b1730 2024-03-25T18:25:35.626 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.624+0000 7f34717fa640 1 -- 172.21.15.155:0/30144527 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2413697321 0 0) 0x7f3458004eb0 con 0x7f347414be90 2024-03-25T18:25:35.626 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.624+0000 7f34717fa640 1 -- 172.21.15.155:0/30144527 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34741d74b0 con 0x7f347414be90 2024-03-25T18:25:35.626 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.624+0000 7f34717fa640 1 -- 172.21.15.155:0/30144527 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f346c001bd0 con 0x7f347414be90 2024-03-25T18:25:35.626 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.624+0000 7f34717fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:35.626 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.624+0000 7f34717fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:35.626 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.624+0000 7f34717fa640 10 monclient(hunting): dump: 2024-03-25T18:25:35.626 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:25:35.626 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:25:35.626 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:35.626 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:35.627 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:35.627 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:25:35.627 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:25:35.627 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:25:35.627 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:25:35.627 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:35.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.634+0000 7f34717fa640 1 -- 172.21.15.155:0/30144527 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1457462512 0 0) 0x7f34741384c0 con 0x7f34740b1730 2024-03-25T18:25:35.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.634+0000 7f34717fa640 1 -- 172.21.15.155:0/30144527 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3458004eb0 con 0x7f34740b1730 2024-03-25T18:25:35.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.634+0000 7f34717fa640 1 -- 172.21.15.155:0/30144527 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f34680015b0 con 0x7f34740b1730 2024-03-25T18:25:35.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.634+0000 7f34717fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:35.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.634+0000 7f34717fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:35.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.634+0000 7f34717fa640 10 monclient(hunting): dump: 2024-03-25T18:25:35.636 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:25:35.637 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:25:35.637 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:35.637 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:35.637 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:35.637 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:25:35.637 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:25:35.637 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:25:35.637 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:25:35.637 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:35.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.636+0000 7f34717fa640 1 -- 172.21.15.155:0/30144527 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1739345225 0 0) 0x7f34741d74b0 con 0x7f347414be90 2024-03-25T18:25:35.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.636+0000 7f34717fa640 1 -- 172.21.15.155:0/30144527 >> v1:172.21.15.167:6789/0 conn(0x7f34740b4d60 legacy=0x7f34741367b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:35.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.636+0000 7f34717fa640 1 -- 172.21.15.155:0/30144527 >> v1:172.21.15.155:6789/0 conn(0x7f34740b1730 legacy=0x7f34741d7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:35.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.636+0000 7f34717fa640 10 monclient: _finish_hunting 0 2024-03-25T18:25:35.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.636+0000 7f34717fa640 1 monclient: found mon.c 2024-03-25T18:25:35.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.636+0000 7f34717fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:35.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.636+0000 7f34717fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:25:35.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.636+0000 7f34717fa640 1 -- 172.21.15.155:0/30144527 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3474143900 con 0x7f347414be90 2024-03-25T18:25:35.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.636+0000 7f34717fa640 10 monclient: _finish_auth 0 2024-03-25T18:25:35.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.636+0000 7f34717fa640 10 monclient: _check_auth_tickets 2024-03-25T18:25:35.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.636+0000 7f34717fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:35.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.636+0000 7f347aca8640 5 monclient: authenticate success, global_id 4262 2024-03-25T18:25:35.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.636+0000 7f347aca8640 10 monclient: _renew_subs 2024-03-25T18:25:35.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.636+0000 7f347aca8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:25:35.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.636+0000 7f347aca8640 1 -- 172.21.15.155:0/30144527 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34741d8050 con 0x7f347414be90 2024-03-25T18:25:35.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.636+0000 7f347aca8640 10 client.4262.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:35.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.636+0000 7f347aca8640 10 monclient: _renew_subs 2024-03-25T18:25:35.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.636+0000 7f347aca8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:25:35.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.636+0000 7f347aca8640 1 -- 172.21.15.155:0/30144527 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3474136ec0 con 0x7f347414be90 2024-03-25T18:25:35.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.636+0000 7f347aca8640 1 librados: init done 2024-03-25T18:25:35.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.637+0000 7f347aca8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:35.641 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.638+0000 7f347aca8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:35.641 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.638+0000 7f347aca8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:25:35.641 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.638+0000 7f347aca8640 1 -- 172.21.15.155:0/30144527 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34740627b0 con 0x7f347414be90 2024-03-25T18:25:35.951 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.948+0000 7f34717fa640 1 -- 172.21.15.155:0/30144527 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f346c001fe0 con 0x7f347414be90 2024-03-25T18:25:35.951 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.948+0000 7f34717fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:35.951 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.948+0000 7f34717fa640 1 -- 172.21.15.155:0/30144527 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f346c003b10 con 0x7f347414be90 2024-03-25T18:25:35.951 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.948+0000 7f34717fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:35.951 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.948+0000 7f34717fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:35.951 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.948+0000 7f34717fa640 10 monclient: dump: 2024-03-25T18:25:35.951 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:25:35.951 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:25:35.951 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:35.951 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:35.952 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:35.952 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:25:35.952 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:25:35.952 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:25:35.952 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:25:35.952 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:35.952 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.948+0000 7f3472ffd640 4 set_mon_vals no callback set 2024-03-25T18:25:35.953 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.949+0000 7f34717fa640 1 -- 172.21.15.155:0/30144527 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7f346c019ae0 con 0x7f347414be90 2024-03-25T18:25:35.953 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.950+0000 7f3478a44640 10 monclient: get_auth_request con 0x7f345806e0c0 auth_method 0 2024-03-25T18:25:35.953 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.950+0000 7f34717fa640 10 client.4262.objecter ms_handle_connect 0x7f345806e0c0 2024-03-25T18:25:35.958 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.956+0000 7f34717fa640 1 -- 172.21.15.155:0/30144527 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7f346c053710 con 0x7f347414be90 2024-03-25T18:25:35.958 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.956+0000 7f34717fa640 10 client.4262.objecter ms_dispatch 0x7f3474000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:35.958 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.956+0000 7f34717fa640 3 client.4262.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:35.958 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.956+0000 7f34717fa640 3 client.4262.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:35.958 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.956+0000 7f34717fa640 20 client.4262.objecter dump_active .. 0 homeless 2024-03-25T18:25:35.958 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.956+0000 7f34717fa640 1 -- 172.21.15.155:0/30144527 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f346c04f050 con 0x7f347414be90 2024-03-25T18:25:35.958 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.956+0000 7f34717fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:35.958 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:35.956+0000 7f34717fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:36.213 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:36.211+0000 7f347aca8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:36.213 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:36.211+0000 7f347aca8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:36.213 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:36.211+0000 7f347aca8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:25:36.213 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:36.211+0000 7f347aca8640 1 -- 172.21.15.155:0/30144527 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f34741d7660 con 0x7f347414be90 2024-03-25T18:25:36.243 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:36.241+0000 7f34717fa640 1 -- 172.21.15.155:0/30144527 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7f34741d7660 con 0x7f347414be90 2024-03-25T18:25:36.243 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:36.241+0000 7f34717fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:36.243 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:36.241+0000 7f34717fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:36.243 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:25:36.243 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:36.246 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:36.243+0000 7f347aca8640 10 librados: watch_flush enter 2024-03-25T18:25:36.246 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:36.243+0000 7f347aca8640 10 librados: watch_flush exit 2024-03-25T18:25:36.246 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:36.243+0000 7f347aca8640 20 client.4262.objecter shutdown clearing up homeless session... 2024-03-25T18:25:36.246 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:36.243+0000 7f347aca8640 10 client.4262.objecter successfully canceled tick 2024-03-25T18:25:36.246 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:36.243+0000 7f347aca8640 1 -- 172.21.15.155:0/30144527 >> v1:172.21.15.167:6800/2919070503 conn(0x7f345806e0c0 legacy=0x7f3458070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:36.246 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:36.243+0000 7f347aca8640 10 monclient: shutdown 2024-03-25T18:25:36.246 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:36.243+0000 7f347aca8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:36.246 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:36.243+0000 7f347aca8640 1 -- 172.21.15.155:0/30144527 >> v1:172.21.15.155:6790/0 conn(0x7f347414be90 legacy=0x7f34741e1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:36.246 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:36.243+0000 7f347aca8640 1 -- 172.21.15.155:0/30144527 shutdown_connections 2024-03-25T18:25:36.246 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:36.243+0000 7f347aca8640 1 -- 172.21.15.155:0/30144527 >> 172.21.15.155:0/30144527 conn(0x7f34740617a0 msgr2=0x7f34740b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:36.246 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:36.243+0000 7f347aca8640 1 -- 172.21.15.155:0/30144527 shutdown_connections 2024-03-25T18:25:36.247 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:36.243+0000 7f347aca8640 1 -- 172.21.15.155:0/30144527 wait complete. 2024-03-25T18:25:36.247 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:36.243+0000 7f347aca8640 1 librados: shutdown 2024-03-25T18:25:36.256 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:40.476 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:01:14.081 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:40.553 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:01:14.158 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:40.585 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:01:14.190 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:40.646 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:01:14.225 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:40.648 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:01:14.253 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:40.752 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:01:14.331 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:40.785 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:01:14.389 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:40.879 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:01:14.484 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:40.990 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:01:14.569 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:43.063 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:43.169 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.169+0000 7efce8c55640 10 monclient: get_monmap_and_config 2024-03-25T18:25:43.170 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.169+0000 7efce8c55640 10 monclient: build_initial_monmap 2024-03-25T18:25:43.170 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.169+0000 7efce8c55640 10 monclient: monmap: 2024-03-25T18:25:43.170 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:25:43.170 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:25:43.170 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:25:43.170026+0000 2024-03-25T18:25:43.170 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:25:43.170026+0000 2024-03-25T18:25:43.170 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:43.170 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:25:43.170 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:25:43.171 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:25:43.171 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:25:43.171 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:43.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.169+0000 7efce8c55640 1 Processor -- start 2024-03-25T18:25:43.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.169+0000 7efce8c55640 1 -- start start 2024-03-25T18:25:43.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.169+0000 7efce8c55640 10 monclient: init 2024-03-25T18:25:43.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.170+0000 7efce8c55640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:43.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.170+0000 7efce8c55640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:25:43.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.170+0000 7efce8c55640 10 monclient(hunting): picked mon.noname-b con 0x7efce40b4d60 addr v1:172.21.15.155:6790/0 2024-03-25T18:25:43.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.170+0000 7efce8c55640 10 monclient(hunting): picked mon.noname-a con 0x7efce40b1730 addr v1:172.21.15.155:6789/0 2024-03-25T18:25:43.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.170+0000 7efce8c55640 10 monclient(hunting): picked mon.noname-c con 0x7efce414be90 addr v1:172.21.15.167:6789/0 2024-03-25T18:25:43.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.170+0000 7efce8c55640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efce40af940 con 0x7efce40b1730 2024-03-25T18:25:43.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.170+0000 7efce8c55640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efce40b0a70 con 0x7efce414be90 2024-03-25T18:25:43.173 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.170+0000 7efce8c55640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efce40b0d50 con 0x7efce40b4d60 2024-03-25T18:25:43.173 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.170+0000 7efce8c55640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:43.173 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.170+0000 7efce8c55640 10 monclient(hunting): authenticate will time out at 7589.480272s 2024-03-25T18:25:43.173 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.171+0000 7efce2d9d640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7efce414be90 0x7efce414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:49270/0 (socket says 172.21.15.155:49270) 2024-03-25T18:25:43.173 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.171+0000 7efce2d9d640 1 -- 172.21.15.155:0/1813311703 learned_addr learned my addr 172.21.15.155:0/1813311703 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:25:43.239 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.238+0000 7efce159a640 1 -- 172.21.15.155:0/1813311703 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3109781459 0 0) 0x7efce40af940 con 0x7efce40b1730 2024-03-25T18:25:43.239 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.238+0000 7efce159a640 10 monclient(hunting): _negotiate 2024-03-25T18:25:43.239 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.238+0000 7efce159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:43.240 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.238+0000 7efce159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:43.240 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.238+0000 7efce159a640 10 monclient(hunting): my global_id is 4263 2024-03-25T18:25:43.240 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.238+0000 7efce159a640 1 -- 172.21.15.155:0/1813311703 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efccc003520 con 0x7efce40b1730 2024-03-25T18:25:43.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.260+0000 7efce159a640 1 -- 172.21.15.155:0/1813311703 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 925274919 0 0) 0x7efccc003520 con 0x7efce40b1730 2024-03-25T18:25:43.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.260+0000 7efce159a640 1 -- 172.21.15.155:0/1813311703 >> v1:172.21.15.155:6790/0 conn(0x7efce40b4d60 legacy=0x7efce413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:43.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.260+0000 7efce159a640 1 -- 172.21.15.155:0/1813311703 >> v1:172.21.15.167:6789/0 conn(0x7efce414be90 legacy=0x7efce414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:43.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.260+0000 7efce159a640 10 monclient: _finish_hunting 0 2024-03-25T18:25:43.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.260+0000 7efce159a640 1 monclient: found mon.noname-a 2024-03-25T18:25:43.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.260+0000 7efce159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:43.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.260+0000 7efce159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-25T18:25:43.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.260+0000 7efce159a640 1 -- 172.21.15.155:0/1813311703 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efce40b1030 con 0x7efce40b1730 2024-03-25T18:25:43.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.260+0000 7efce159a640 10 monclient: _finish_auth 0 2024-03-25T18:25:43.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.260+0000 7efce159a640 10 monclient: _check_auth_tickets 2024-03-25T18:25:43.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.260+0000 7efce159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:43.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.260+0000 7efce159a640 1 -- 172.21.15.155:0/1813311703 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7efcd4001690 con 0x7efce40b1730 2024-03-25T18:25:43.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.260+0000 7efce8c55640 5 monclient: authenticate success, global_id 4263 2024-03-25T18:25:43.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.260+0000 7efce8c55640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:43.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.260+0000 7efce159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:43.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.260+0000 7efce159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:43.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.260+0000 7efce159a640 10 monclient: dump: 2024-03-25T18:25:43.263 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:25:43.263 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:25:43.263 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:43.263 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:43.264 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:43.264 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:25:43.264 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:25:43.264 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:25:43.264 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:25:43.264 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:43.264 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.260+0000 7efce8c55640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:43.295 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.294+0000 7efce159a640 1 -- 172.21.15.155:0/1813311703 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efcd40017f0 con 0x7efce40b1730 2024-03-25T18:25:43.295 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.294+0000 7efce159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:43.296 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.294+0000 7efce159a640 1 -- 172.21.15.155:0/1813311703 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7efcd4001c00 con 0x7efce40b1730 2024-03-25T18:25:43.296 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.294+0000 7efce8c55640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:43.296 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.294+0000 7efce8c55640 4 set_mon_vals no callback set 2024-03-25T18:25:43.296 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.294+0000 7efce159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:43.296 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.294+0000 7efce159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:43.296 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.294+0000 7efce159a640 10 monclient: dump: 2024-03-25T18:25:43.296 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:25:43.296 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:25:43.296 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:43.296 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:43.296 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:43.296 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:25:43.296 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:25:43.296 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:25:43.297 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:25:43.297 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:43.299 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.295+0000 7efce8c55640 10 monclient: shutdown 2024-03-25T18:25:43.299 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.295+0000 7efce8c55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:43.299 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.295+0000 7efce8c55640 1 -- 172.21.15.155:0/1813311703 >> v1:172.21.15.155:6789/0 conn(0x7efce40b1730 legacy=0x7efce40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:43.299 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.295+0000 7efce8c55640 1 -- 172.21.15.155:0/1813311703 shutdown_connections 2024-03-25T18:25:43.299 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.295+0000 7efce8c55640 1 -- 172.21.15.155:0/1813311703 >> 172.21.15.155:0/1813311703 conn(0x7efce40617a0 msgr2=0x7efce4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:43.299 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.295+0000 7efce8c55640 1 -- 172.21.15.155:0/1813311703 shutdown_connections 2024-03-25T18:25:43.299 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.295+0000 7efce8c55640 1 -- 172.21.15.155:0/1813311703 wait complete. 2024-03-25T18:25:43.299 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.296+0000 7efce8c55640 10 monclient: build_initial_monmap 2024-03-25T18:25:43.300 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.296+0000 7efce8c55640 10 monclient: monmap: 2024-03-25T18:25:43.300 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:25:43.300 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:43.300 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:25:43.300 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:25:43.300 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:43.300 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:25:43.300 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:25:43.300 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:25:43.300 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:25:43.300 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:43.300 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.296+0000 7efce8c55640 1 librados: starting msgr at 2024-03-25T18:25:43.300 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.296+0000 7efce8c55640 1 librados: starting objecter 2024-03-25T18:25:43.300 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.297+0000 7efce8c55640 1 Processor -- start 2024-03-25T18:25:43.300 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.297+0000 7efce8c55640 1 -- start start 2024-03-25T18:25:43.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.297+0000 7efce8c55640 1 librados: setting wanted keys 2024-03-25T18:25:43.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.297+0000 7efce8c55640 1 librados: calling monclient init 2024-03-25T18:25:43.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.297+0000 7efce8c55640 10 monclient: init 2024-03-25T18:25:43.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.297+0000 7efce8c55640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:43.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.297+0000 7efce8c55640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:25:43.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.297+0000 7efce8c55640 10 monclient(hunting): picked mon.noname-c con 0x7efce40b1730 addr v1:172.21.15.155:6790/0 2024-03-25T18:25:43.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.297+0000 7efce8c55640 10 monclient(hunting): picked mon.noname-b con 0x7efce40b4d60 addr v1:172.21.15.167:6789/0 2024-03-25T18:25:43.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.297+0000 7efce8c55640 10 monclient(hunting): picked mon.noname-a con 0x7efce414be90 addr v1:172.21.15.155:6789/0 2024-03-25T18:25:43.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.297+0000 7efce8c55640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efce41420d0 con 0x7efce414be90 2024-03-25T18:25:43.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.298+0000 7efce8c55640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efce4138270 con 0x7efce40b4d60 2024-03-25T18:25:43.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.298+0000 7efce8c55640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efce4138400 con 0x7efce40b1730 2024-03-25T18:25:43.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.298+0000 7efce8c55640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:43.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.298+0000 7efce8c55640 10 monclient(hunting): authenticate will time out at 7589.607815s 2024-03-25T18:25:43.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.307+0000 7efce1d9b640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7efce40b4d60 0x7efce41405f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:49322/0 (socket says 172.21.15.155:49322) 2024-03-25T18:25:43.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.307+0000 7efce1d9b640 1 -- 172.21.15.155:0/2396000250 learned_addr learned my addr 172.21.15.155:0/2396000250 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:25:43.310 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.309+0000 7efcd2ffd640 10 client.?.objecter ms_handle_connect 0x7efce40b4d60 2024-03-25T18:25:43.310 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.309+0000 7efcd2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:43.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.317+0000 7efcd2ffd640 10 client.?.objecter ms_handle_connect 0x7efce40b1730 2024-03-25T18:25:43.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.317+0000 7efcd2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:43.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.318+0000 7efcd2ffd640 1 -- 172.21.15.155:0/2396000250 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3280410769 0 0) 0x7efce4138270 con 0x7efce40b4d60 2024-03-25T18:25:43.319 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.318+0000 7efcd2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:43.319 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.318+0000 7efcd2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:43.319 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.318+0000 7efcd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:43.319 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.318+0000 7efcd2ffd640 10 monclient(hunting): my global_id is 4267 2024-03-25T18:25:43.319 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.318+0000 7efcd2ffd640 1 -- 172.21.15.155:0/2396000250 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcc4004eb0 con 0x7efce40b4d60 2024-03-25T18:25:43.320 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.320+0000 7efcd2ffd640 10 client.?.objecter ms_handle_connect 0x7efce414be90 2024-03-25T18:25:43.321 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.320+0000 7efcd2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:43.334 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.333+0000 7efcd2ffd640 1 -- 172.21.15.155:0/2396000250 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 992159410 0 0) 0x7efce4138400 con 0x7efce40b1730 2024-03-25T18:25:43.334 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.333+0000 7efcd2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:43.334 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.333+0000 7efcd2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:43.335 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.333+0000 7efcd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:43.335 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.333+0000 7efcd2ffd640 10 monclient(hunting): my global_id is 4268 2024-03-25T18:25:43.335 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.333+0000 7efcd2ffd640 1 -- 172.21.15.155:0/2396000250 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efce4138270 con 0x7efce40b1730 2024-03-25T18:25:43.339 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.338+0000 7efcd2ffd640 1 -- 172.21.15.155:0/2396000250 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2678879516 0 0) 0x7efcc4004eb0 con 0x7efce40b4d60 2024-03-25T18:25:43.339 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.338+0000 7efcd2ffd640 1 -- 172.21.15.155:0/2396000250 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efce4138400 con 0x7efce40b4d60 2024-03-25T18:25:43.339 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.338+0000 7efcd2ffd640 1 -- 172.21.15.155:0/2396000250 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7efcd4001690 con 0x7efce40b4d60 2024-03-25T18:25:43.339 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.338+0000 7efcd2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:43.339 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.338+0000 7efcd2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:25:43.339 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.338+0000 7efcd2ffd640 10 monclient(hunting): dump: 2024-03-25T18:25:43.339 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:25:43.339 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:25:43.339 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:43.339 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:43.339 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:43.339 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:25:43.340 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:25:43.340 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:25:43.340 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:25:43.340 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:43.340 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.338+0000 7efcd2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:43.340 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.339+0000 7efcd2ffd640 1 -- 172.21.15.155:0/2396000250 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2673647206 0 0) 0x7efce41420d0 con 0x7efce414be90 2024-03-25T18:25:43.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.339+0000 7efcd2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:43.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.339+0000 7efcd2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:43.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.339+0000 7efcd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:43.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.339+0000 7efcd2ffd640 10 monclient(hunting): my global_id is 4266 2024-03-25T18:25:43.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.339+0000 7efcd2ffd640 1 -- 172.21.15.155:0/2396000250 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcc4004eb0 con 0x7efce414be90 2024-03-25T18:25:43.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.351+0000 7efcd2ffd640 1 -- 172.21.15.155:0/2396000250 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1333310451 0 0) 0x7efcc4004eb0 con 0x7efce414be90 2024-03-25T18:25:43.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.351+0000 7efcd2ffd640 1 -- 172.21.15.155:0/2396000250 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efce41420d0 con 0x7efce414be90 2024-03-25T18:25:43.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.351+0000 7efcd2ffd640 1 -- 172.21.15.155:0/2396000250 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7efcdc001a60 con 0x7efce414be90 2024-03-25T18:25:43.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.351+0000 7efcd2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:43.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.351+0000 7efcd2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:43.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.351+0000 7efcd2ffd640 10 monclient(hunting): dump: 2024-03-25T18:25:43.352 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:25:43.352 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:25:43.352 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:43.353 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:43.353 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:43.353 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:25:43.353 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:25:43.353 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:25:43.353 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:25:43.353 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:43.359 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.358+0000 7efcd2ffd640 1 -- 172.21.15.155:0/2396000250 <== mon.1 v1:172.21.15.167:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3394023861 0 0) 0x7efce4138400 con 0x7efce40b4d60 2024-03-25T18:25:43.359 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.358+0000 7efcd2ffd640 1 -- 172.21.15.155:0/2396000250 >> v1:172.21.15.155:6790/0 conn(0x7efce40b1730 legacy=0x7efce41db1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:43.359 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.358+0000 7efcd2ffd640 1 -- 172.21.15.155:0/2396000250 >> v1:172.21.15.155:6789/0 conn(0x7efce414be90 legacy=0x7efce4134d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:43.359 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.358+0000 7efcd2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:25:43.359 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.358+0000 7efcd2ffd640 1 monclient: found mon.b 2024-03-25T18:25:43.359 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.358+0000 7efcd2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:43.359 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.358+0000 7efcd2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:25:43.359 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.358+0000 7efcd2ffd640 1 -- 172.21.15.155:0/2396000250 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efce4142480 con 0x7efce40b4d60 2024-03-25T18:25:43.359 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.358+0000 7efcd2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:25:43.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.358+0000 7efcd2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:25:43.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.358+0000 7efcd2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:43.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.358+0000 7efce8c55640 5 monclient: authenticate success, global_id 4267 2024-03-25T18:25:43.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.358+0000 7efce8c55640 10 monclient: _renew_subs 2024-03-25T18:25:43.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.358+0000 7efce8c55640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:25:43.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.358+0000 7efce8c55640 1 -- 172.21.15.155:0/2396000250 --> v1:172.21.15.167:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efce41426d0 con 0x7efce40b4d60 2024-03-25T18:25:43.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.358+0000 7efce8c55640 10 client.4267.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:43.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.358+0000 7efce8c55640 10 monclient: _renew_subs 2024-03-25T18:25:43.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.358+0000 7efce8c55640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:25:43.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.358+0000 7efce8c55640 1 -- 172.21.15.155:0/2396000250 --> v1:172.21.15.167:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efce41dba90 con 0x7efce40b4d60 2024-03-25T18:25:43.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.358+0000 7efce8c55640 1 librados: init done 2024-03-25T18:25:43.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.359+0000 7efce8c55640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:43.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.359+0000 7efce8c55640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:43.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.359+0000 7efce8c55640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:25:43.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.359+0000 7efce8c55640 1 -- 172.21.15.155:0/2396000250 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efce41401f0 con 0x7efce40b4d60 2024-03-25T18:25:43.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.666+0000 7efcd2ffd640 1 -- 172.21.15.155:0/2396000250 <== mon.1 v1:172.21.15.167:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efcd40038a0 con 0x7efce40b4d60 2024-03-25T18:25:43.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.666+0000 7efcd2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:43.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.666+0000 7efcd2ffd640 1 -- 172.21.15.155:0/2396000250 <== mon.1 v1:172.21.15.167:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7efcd4003c70 con 0x7efce40b4d60 2024-03-25T18:25:43.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.666+0000 7efcd2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:43.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.666+0000 7efcd2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:43.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.666+0000 7efcd2ffd640 10 monclient: dump: 2024-03-25T18:25:43.669 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:25:43.669 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:25:43.669 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:43.669 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:43.669 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:43.670 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:25:43.670 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:25:43.670 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:25:43.670 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:25:43.670 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:43.670 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.666+0000 7efce0d99640 4 set_mon_vals no callback set 2024-03-25T18:25:43.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.667+0000 7efcd2ffd640 1 -- 172.21.15.155:0/2396000250 <== mon.1 v1:172.21.15.167:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7efcd4003e90 con 0x7efce40b4d60 2024-03-25T18:25:43.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.668+0000 7efce259c640 10 monclient: get_auth_request con 0x7efcc406dfb0 auth_method 0 2024-03-25T18:25:43.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.668+0000 7efcd2ffd640 10 client.4267.objecter ms_handle_connect 0x7efcc406dfb0 2024-03-25T18:25:43.673 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.672+0000 7efcd2ffd640 1 -- 172.21.15.155:0/2396000250 <== mon.1 v1:172.21.15.167:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7efcd4054980 con 0x7efce40b4d60 2024-03-25T18:25:43.673 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.672+0000 7efcd2ffd640 10 client.4267.objecter ms_dispatch 0x7efce4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:43.673 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.672+0000 7efcd2ffd640 3 client.4267.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:43.674 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.672+0000 7efcd2ffd640 3 client.4267.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:43.674 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.672+0000 7efcd2ffd640 20 client.4267.objecter dump_active .. 0 homeless 2024-03-25T18:25:43.674 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.673+0000 7efcd2ffd640 1 -- 172.21.15.155:0/2396000250 <== mon.1 v1:172.21.15.167:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efcd4050490 con 0x7efce40b4d60 2024-03-25T18:25:43.675 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.673+0000 7efcd2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:43.675 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.673+0000 7efcd2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:43.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.924+0000 7efce8c55640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:43.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.924+0000 7efce8c55640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:43.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.924+0000 7efce8c55640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:25:43.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.924+0000 7efce8c55640 1 -- 172.21.15.155:0/2396000250 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efce41dae70 con 0x7efce40b4d60 2024-03-25T18:25:43.953 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.952+0000 7efcd2ffd640 1 -- 172.21.15.155:0/2396000250 <== mon.1 v1:172.21.15.167:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7efcd40523c0 con 0x7efce40b4d60 2024-03-25T18:25:43.953 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.952+0000 7efcd2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:43.954 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.952+0000 7efcd2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:43.954 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:25:43.954 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:43.956 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.954+0000 7efce8c55640 10 librados: watch_flush enter 2024-03-25T18:25:43.956 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.954+0000 7efce8c55640 10 librados: watch_flush exit 2024-03-25T18:25:43.956 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.954+0000 7efce8c55640 20 client.4267.objecter shutdown clearing up homeless session... 2024-03-25T18:25:43.956 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.954+0000 7efce8c55640 10 client.4267.objecter successfully canceled tick 2024-03-25T18:25:43.956 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.955+0000 7efce8c55640 1 -- 172.21.15.155:0/2396000250 >> v1:172.21.15.167:6800/2919070503 conn(0x7efcc406dfb0 legacy=0x7efcc4070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:43.956 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.955+0000 7efce8c55640 10 monclient: shutdown 2024-03-25T18:25:43.957 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.955+0000 7efce8c55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:43.957 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.955+0000 7efce8c55640 1 -- 172.21.15.155:0/2396000250 >> v1:172.21.15.167:6789/0 conn(0x7efce40b4d60 legacy=0x7efce41405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:43.957 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.955+0000 7efce8c55640 1 -- 172.21.15.155:0/2396000250 shutdown_connections 2024-03-25T18:25:43.957 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.955+0000 7efce8c55640 1 -- 172.21.15.155:0/2396000250 >> 172.21.15.155:0/2396000250 conn(0x7efce40617a0 msgr2=0x7efce40b2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:43.957 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.955+0000 7efce8c55640 1 -- 172.21.15.155:0/2396000250 shutdown_connections 2024-03-25T18:25:43.957 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.955+0000 7efce8c55640 1 -- 172.21.15.155:0/2396000250 wait complete. 2024-03-25T18:25:43.957 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:43.955+0000 7efce8c55640 1 librados: shutdown 2024-03-25T18:25:43.968 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:50.775 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:50.889 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.888+0000 7fde7629a640 10 monclient: get_monmap_and_config 2024-03-25T18:25:50.889 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.888+0000 7fde7629a640 10 monclient: build_initial_monmap 2024-03-25T18:25:50.889 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.888+0000 7fde7629a640 10 monclient: monmap: 2024-03-25T18:25:50.889 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:25:50.889 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:25:50.889 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:25:50.889255+0000 2024-03-25T18:25:50.890 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:25:50.889255+0000 2024-03-25T18:25:50.890 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:50.890 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:25:50.890 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:25:50.890 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:25:50.890 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:25:50.890 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:50.892 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.889+0000 7fde7629a640 1 Processor -- start 2024-03-25T18:25:50.892 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.889+0000 7fde7629a640 1 -- start start 2024-03-25T18:25:50.892 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.889+0000 7fde7629a640 10 monclient: init 2024-03-25T18:25:50.892 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.890+0000 7fde7629a640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:50.892 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.890+0000 7fde7629a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:25:50.892 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.890+0000 7fde7629a640 10 monclient(hunting): picked mon.noname-c con 0x7fde700a98f0 addr v1:172.21.15.167:6789/0 2024-03-25T18:25:50.892 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.890+0000 7fde7629a640 10 monclient(hunting): picked mon.noname-a con 0x7fde7013f710 addr v1:172.21.15.155:6789/0 2024-03-25T18:25:50.892 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.890+0000 7fde7629a640 10 monclient(hunting): picked mon.noname-b con 0x7fde701430c0 addr v1:172.21.15.155:6790/0 2024-03-25T18:25:50.892 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.890+0000 7fde7629a640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde70142520 con 0x7fde7013f710 2024-03-25T18:25:50.892 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.890+0000 7fde7629a640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde700b28a0 con 0x7fde700a98f0 2024-03-25T18:25:50.892 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.890+0000 7fde7629a640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde700b2a30 con 0x7fde701430c0 2024-03-25T18:25:50.892 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.890+0000 7fde7629a640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:50.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.890+0000 7fde7629a640 10 monclient(hunting): authenticate will time out at 7597.199993s 2024-03-25T18:25:50.895 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.895+0000 7fde74837640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fde701430c0 0x7fde7014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:58742/0 (socket says 172.21.15.155:58742) 2024-03-25T18:25:50.896 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.895+0000 7fde74837640 1 -- 172.21.15.155:0/1993703811 learned_addr learned my addr 172.21.15.155:0/1993703811 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:25:50.908 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.907+0000 7fde6f7fe640 1 -- 172.21.15.155:0/1993703811 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3608823588 0 0) 0x7fde700b2a30 con 0x7fde701430c0 2024-03-25T18:25:50.908 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.907+0000 7fde6f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:50.908 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.907+0000 7fde6f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:50.909 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.907+0000 7fde6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:50.909 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.907+0000 7fde6f7fe640 10 monclient(hunting): my global_id is 4271 2024-03-25T18:25:50.909 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.907+0000 7fde6f7fe640 1 -- 172.21.15.155:0/1993703811 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde5c003520 con 0x7fde701430c0 2024-03-25T18:25:50.920 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.919+0000 7fde6f7fe640 1 -- 172.21.15.155:0/1993703811 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1407709331 0 0) 0x7fde700b28a0 con 0x7fde700a98f0 2024-03-25T18:25:50.920 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.919+0000 7fde6f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:50.920 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.919+0000 7fde6f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:50.920 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.919+0000 7fde6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:50.920 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.919+0000 7fde6f7fe640 10 monclient(hunting): my global_id is 4270 2024-03-25T18:25:50.921 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.919+0000 7fde6f7fe640 1 -- 172.21.15.155:0/1993703811 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde700b2a30 con 0x7fde700a98f0 2024-03-25T18:25:50.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.926+0000 7fde6f7fe640 1 -- 172.21.15.155:0/1993703811 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2654036545 0 0) 0x7fde70142520 con 0x7fde7013f710 2024-03-25T18:25:50.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.926+0000 7fde6f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:50.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.926+0000 7fde6f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:50.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.926+0000 7fde6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:50.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.926+0000 7fde6f7fe640 10 monclient(hunting): my global_id is 4269 2024-03-25T18:25:50.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.926+0000 7fde6f7fe640 1 -- 172.21.15.155:0/1993703811 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde700b28a0 con 0x7fde7013f710 2024-03-25T18:25:50.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.926+0000 7fde6f7fe640 1 -- 172.21.15.155:0/1993703811 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2806477359 0 0) 0x7fde5c003520 con 0x7fde701430c0 2024-03-25T18:25:50.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.926+0000 7fde6f7fe640 1 -- 172.21.15.155:0/1993703811 >> v1:172.21.15.167:6789/0 conn(0x7fde700a98f0 legacy=0x7fde700a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:50.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.926+0000 7fde6f7fe640 1 -- 172.21.15.155:0/1993703811 >> v1:172.21.15.155:6789/0 conn(0x7fde7013f710 legacy=0x7fde701429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:50.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.926+0000 7fde6f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:25:50.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.926+0000 7fde6f7fe640 1 monclient: found mon.noname-b 2024-03-25T18:25:50.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.926+0000 7fde6f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:50.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.926+0000 7fde6f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-25T18:25:50.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.926+0000 7fde6f7fe640 1 -- 172.21.15.155:0/1993703811 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde700b44c0 con 0x7fde701430c0 2024-03-25T18:25:50.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.926+0000 7fde6f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:25:50.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.926+0000 7fde6f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:25:50.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.926+0000 7fde6f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:50.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.926+0000 7fde6f7fe640 1 -- 172.21.15.155:0/1993703811 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fde68001a10 con 0x7fde701430c0 2024-03-25T18:25:50.929 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.926+0000 7fde6f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:50.929 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.926+0000 7fde6f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:25:50.929 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.926+0000 7fde6f7fe640 10 monclient: dump: 2024-03-25T18:25:50.929 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:25:50.929 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:25:50.929 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:50.929 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:50.929 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:50.929 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:25:50.929 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:25:50.929 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:25:50.929 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:25:50.929 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:50.929 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.926+0000 7fde7629a640 5 monclient: authenticate success, global_id 4271 2024-03-25T18:25:50.929 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.926+0000 7fde7629a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:50.941 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.940+0000 7fde6f7fe640 1 -- 172.21.15.155:0/1993703811 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fde68001bf0 con 0x7fde701430c0 2024-03-25T18:25:50.941 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.940+0000 7fde6f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:50.941 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.940+0000 7fde6f7fe640 1 -- 172.21.15.155:0/1993703811 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fde68001f80 con 0x7fde701430c0 2024-03-25T18:25:50.941 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.940+0000 7fde7629a640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:50.941 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.940+0000 7fde7629a640 4 set_mon_vals no callback set 2024-03-25T18:25:50.941 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.940+0000 7fde6f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:50.941 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.940+0000 7fde6f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:50.941 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.940+0000 7fde6f7fe640 10 monclient: dump: 2024-03-25T18:25:50.941 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:25:50.941 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:25:50.942 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:50.942 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:50.942 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:50.942 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:25:50.942 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:25:50.942 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:25:50.942 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:25:50.942 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:50.944 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.941+0000 7fde7629a640 10 monclient: shutdown 2024-03-25T18:25:50.944 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.941+0000 7fde7629a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:50.944 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.941+0000 7fde7629a640 1 -- 172.21.15.155:0/1993703811 >> v1:172.21.15.155:6790/0 conn(0x7fde701430c0 legacy=0x7fde7014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:50.944 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.941+0000 7fde7629a640 1 -- 172.21.15.155:0/1993703811 shutdown_connections 2024-03-25T18:25:50.944 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.941+0000 7fde7629a640 1 -- 172.21.15.155:0/1993703811 >> 172.21.15.155:0/1993703811 conn(0x7fde700b7ea0 msgr2=0x7fde700b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:50.945 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.941+0000 7fde7629a640 1 -- 172.21.15.155:0/1993703811 shutdown_connections 2024-03-25T18:25:50.945 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.941+0000 7fde7629a640 1 -- 172.21.15.155:0/1993703811 wait complete. 2024-03-25T18:25:50.945 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.942+0000 7fde7629a640 10 monclient: build_initial_monmap 2024-03-25T18:25:50.945 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.942+0000 7fde7629a640 10 monclient: monmap: 2024-03-25T18:25:50.945 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:25:50.945 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:50.945 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:25:50.945 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:25:50.945 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:50.945 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:25:50.945 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:25:50.945 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:25:50.945 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:25:50.945 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:50.945 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.942+0000 7fde7629a640 1 librados: starting msgr at 2024-03-25T18:25:50.945 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.942+0000 7fde7629a640 1 librados: starting objecter 2024-03-25T18:25:50.945 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.942+0000 7fde7629a640 1 Processor -- start 2024-03-25T18:25:50.946 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.942+0000 7fde7629a640 1 -- start start 2024-03-25T18:25:50.946 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.942+0000 7fde7629a640 1 librados: setting wanted keys 2024-03-25T18:25:50.946 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.942+0000 7fde7629a640 1 librados: calling monclient init 2024-03-25T18:25:50.946 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.942+0000 7fde7629a640 10 monclient: init 2024-03-25T18:25:50.946 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.943+0000 7fde7629a640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:50.946 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.943+0000 7fde7629a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:25:50.946 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.943+0000 7fde7629a640 10 monclient(hunting): picked mon.noname-c con 0x7fde700a98f0 addr v1:172.21.15.155:6790/0 2024-03-25T18:25:50.946 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.943+0000 7fde7629a640 10 monclient(hunting): picked mon.noname-b con 0x7fde7013f710 addr v1:172.21.15.167:6789/0 2024-03-25T18:25:50.946 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.943+0000 7fde7629a640 10 monclient(hunting): picked mon.noname-a con 0x7fde701430c0 addr v1:172.21.15.155:6789/0 2024-03-25T18:25:50.946 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.943+0000 7fde7629a640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde7013c9b0 con 0x7fde701430c0 2024-03-25T18:25:50.946 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.943+0000 7fde7629a640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde701397c0 con 0x7fde7013f710 2024-03-25T18:25:50.946 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.943+0000 7fde7629a640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde70139950 con 0x7fde700a98f0 2024-03-25T18:25:50.946 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.943+0000 7fde7629a640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:50.947 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.943+0000 7fde7629a640 10 monclient(hunting): authenticate will time out at 7597.253079s 2024-03-25T18:25:50.953 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.952+0000 7fde67fff640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7fde7013f710 0x7fde701dae50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:47042/0 (socket says 172.21.15.155:47042) 2024-03-25T18:25:50.953 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.952+0000 7fde67fff640 1 -- 172.21.15.155:0/2443924018 learned_addr learned my addr 172.21.15.155:0/2443924018 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:25:50.956 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.955+0000 7fde6d7fa640 10 client.?.objecter ms_handle_connect 0x7fde7013f710 2024-03-25T18:25:50.956 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.955+0000 7fde6d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:50.957 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.956+0000 7fde6d7fa640 10 client.?.objecter ms_handle_connect 0x7fde700a98f0 2024-03-25T18:25:50.957 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.956+0000 7fde6d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:50.962 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.962+0000 7fde6d7fa640 1 -- 172.21.15.155:0/2443924018 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 622319187 0 0) 0x7fde701397c0 con 0x7fde7013f710 2024-03-25T18:25:50.963 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.962+0000 7fde6d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:25:50.963 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.962+0000 7fde6d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:50.963 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.962+0000 7fde6d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:50.963 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.962+0000 7fde6d7fa640 10 monclient(hunting): my global_id is 4273 2024-03-25T18:25:50.963 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.962+0000 7fde6d7fa640 1 -- 172.21.15.155:0/2443924018 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde50004eb0 con 0x7fde7013f710 2024-03-25T18:25:50.967 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.966+0000 7fde6d7fa640 10 client.?.objecter ms_handle_connect 0x7fde701430c0 2024-03-25T18:25:50.967 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.966+0000 7fde6d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:50.976 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.975+0000 7fde6d7fa640 1 -- 172.21.15.155:0/2443924018 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 552177436 0 0) 0x7fde7013c9b0 con 0x7fde701430c0 2024-03-25T18:25:50.976 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.975+0000 7fde6d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:25:50.976 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.975+0000 7fde6d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:50.976 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.975+0000 7fde6d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:50.976 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.975+0000 7fde6d7fa640 10 monclient(hunting): my global_id is 4272 2024-03-25T18:25:50.976 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.975+0000 7fde6d7fa640 1 -- 172.21.15.155:0/2443924018 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde701397c0 con 0x7fde701430c0 2024-03-25T18:25:50.977 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.976+0000 7fde6d7fa640 1 -- 172.21.15.155:0/2443924018 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2830593257 0 0) 0x7fde50004eb0 con 0x7fde7013f710 2024-03-25T18:25:50.977 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.976+0000 7fde6d7fa640 1 -- 172.21.15.155:0/2443924018 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fde7013c9b0 con 0x7fde7013f710 2024-03-25T18:25:50.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.976+0000 7fde6d7fa640 1 -- 172.21.15.155:0/2443924018 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fde58001a10 con 0x7fde7013f710 2024-03-25T18:25:50.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.976+0000 7fde6d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:50.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.976+0000 7fde6d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:25:50.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.976+0000 7fde6d7fa640 10 monclient(hunting): dump: 2024-03-25T18:25:50.978 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:25:50.979 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:25:50.979 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:50.979 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:50.979 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:50.979 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:25:50.979 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:25:50.979 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:25:50.979 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:25:50.979 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:50.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.976+0000 7fde6d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:50.981 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.981+0000 7fde6d7fa640 1 -- 172.21.15.155:0/2443924018 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3370689162 0 0) 0x7fde70139950 con 0x7fde700a98f0 2024-03-25T18:25:50.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.981+0000 7fde6d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:25:50.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.981+0000 7fde6d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:50.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.981+0000 7fde6d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:50.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.981+0000 7fde6d7fa640 10 monclient(hunting): my global_id is 4274 2024-03-25T18:25:50.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.981+0000 7fde6d7fa640 1 -- 172.21.15.155:0/2443924018 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde50004eb0 con 0x7fde700a98f0 2024-03-25T18:25:50.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.989+0000 7fde6d7fa640 1 -- 172.21.15.155:0/2443924018 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1598101018 0 0) 0x7fde50004eb0 con 0x7fde700a98f0 2024-03-25T18:25:50.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.989+0000 7fde6d7fa640 1 -- 172.21.15.155:0/2443924018 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fde70139950 con 0x7fde700a98f0 2024-03-25T18:25:50.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.989+0000 7fde6d7fa640 1 -- 172.21.15.155:0/2443924018 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fde60001930 con 0x7fde700a98f0 2024-03-25T18:25:50.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.989+0000 7fde6d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:50.991 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.989+0000 7fde6d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:50.991 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.989+0000 7fde6d7fa640 10 monclient(hunting): dump: 2024-03-25T18:25:50.991 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:25:50.991 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:25:50.991 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:50.991 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:50.991 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:50.991 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:25:50.991 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:25:50.991 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:25:50.991 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:25:50.991 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:50.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.990+0000 7fde6d7fa640 1 -- 172.21.15.155:0/2443924018 <== mon.1 v1:172.21.15.167:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3491919647 0 0) 0x7fde7013c9b0 con 0x7fde7013f710 2024-03-25T18:25:50.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.990+0000 7fde6d7fa640 1 -- 172.21.15.155:0/2443924018 >> v1:172.21.15.155:6790/0 conn(0x7fde700a98f0 legacy=0x7fde70130f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:50.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.990+0000 7fde6d7fa640 1 -- 172.21.15.155:0/2443924018 >> v1:172.21.15.155:6789/0 conn(0x7fde701430c0 legacy=0x7fde701e2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:50.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.990+0000 7fde6d7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:25:50.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.990+0000 7fde6d7fa640 1 monclient: found mon.b 2024-03-25T18:25:50.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.991+0000 7fde6d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:50.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.991+0000 7fde6d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:25:50.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.991+0000 7fde6d7fa640 1 -- 172.21.15.155:0/2443924018 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde70131680 con 0x7fde7013f710 2024-03-25T18:25:50.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.991+0000 7fde6d7fa640 10 monclient: _finish_auth 0 2024-03-25T18:25:50.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.991+0000 7fde6d7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:25:50.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.991+0000 7fde6d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:50.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.991+0000 7fde7629a640 5 monclient: authenticate success, global_id 4273 2024-03-25T18:25:50.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.991+0000 7fde7629a640 10 monclient: _renew_subs 2024-03-25T18:25:50.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.991+0000 7fde7629a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:25:50.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.991+0000 7fde7629a640 1 -- 172.21.15.155:0/2443924018 --> v1:172.21.15.167:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde70131950 con 0x7fde7013f710 2024-03-25T18:25:50.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.991+0000 7fde7629a640 10 client.4273.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:50.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.991+0000 7fde7629a640 10 monclient: _renew_subs 2024-03-25T18:25:50.995 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.991+0000 7fde7629a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:25:50.995 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.991+0000 7fde7629a640 1 -- 172.21.15.155:0/2443924018 --> v1:172.21.15.167:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fde70135a30 con 0x7fde7013f710 2024-03-25T18:25:50.995 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.991+0000 7fde7629a640 1 librados: init done 2024-03-25T18:25:50.995 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.991+0000 7fde7629a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:50.995 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.991+0000 7fde7629a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:50.995 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.992+0000 7fde7629a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:25:50.995 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:50.992+0000 7fde7629a640 1 -- 172.21.15.155:0/2443924018 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde7013c810 con 0x7fde7013f710 2024-03-25T18:25:51.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:51.280+0000 7fde6d7fa640 1 -- 172.21.15.155:0/2443924018 <== mon.1 v1:172.21.15.167:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fde58001c70 con 0x7fde7013f710 2024-03-25T18:25:51.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:51.280+0000 7fde6d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:51.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:51.280+0000 7fde6d7fa640 1 -- 172.21.15.155:0/2443924018 <== mon.1 v1:172.21.15.167:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fde580038c0 con 0x7fde7013f710 2024-03-25T18:25:51.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:51.280+0000 7fde6d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:51.282 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:51.280+0000 7fde6d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:51.282 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:51.280+0000 7fde6d7fa640 10 monclient: dump: 2024-03-25T18:25:51.282 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:25:51.282 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:25:51.282 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:51.282 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:51.283 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:51.283 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:25:51.283 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:25:51.283 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:25:51.283 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:25:51.283 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:51.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:51.280+0000 7fde6effd640 4 set_mon_vals no callback set 2024-03-25T18:25:51.285 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:51.281+0000 7fde6d7fa640 1 -- 172.21.15.155:0/2443924018 <== mon.1 v1:172.21.15.167:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7fde58003ae0 con 0x7fde7013f710 2024-03-25T18:25:51.285 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:51.281+0000 7fde6ffff640 10 monclient: get_auth_request con 0x7fde5006dfb0 auth_method 0 2024-03-25T18:25:51.285 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:51.281+0000 7fde6d7fa640 10 client.4273.objecter ms_handle_connect 0x7fde5006dfb0 2024-03-25T18:25:51.286 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:51.285+0000 7fde6d7fa640 1 -- 172.21.15.155:0/2443924018 <== mon.1 v1:172.21.15.167:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7fde58054660 con 0x7fde7013f710 2024-03-25T18:25:51.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:51.285+0000 7fde6d7fa640 10 client.4273.objecter ms_dispatch 0x7fde70000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:51.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:51.285+0000 7fde6d7fa640 3 client.4273.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:51.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:51.285+0000 7fde6d7fa640 3 client.4273.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:51.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:51.285+0000 7fde6d7fa640 20 client.4273.objecter dump_active .. 0 homeless 2024-03-25T18:25:51.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:51.286+0000 7fde6d7fa640 1 -- 172.21.15.155:0/2443924018 <== mon.1 v1:172.21.15.167:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fde58050170 con 0x7fde7013f710 2024-03-25T18:25:51.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:51.286+0000 7fde6d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:51.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:51.286+0000 7fde6d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:51.530 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:51.529+0000 7fde7629a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:51.530 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:51.529+0000 7fde7629a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:51.530 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:51.529+0000 7fde7629a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:25:51.530 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:51.529+0000 7fde7629a640 1 -- 172.21.15.155:0/2443924018 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fde70136fc0 con 0x7fde7013f710 2024-03-25T18:25:51.607 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:51.606+0000 7fde6d7fa640 1 -- 172.21.15.155:0/2443924018 <== mon.1 v1:172.21.15.167:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7fde580520a0 con 0x7fde7013f710 2024-03-25T18:25:51.607 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:51.606+0000 7fde6d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:51.607 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:51.606+0000 7fde6d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:51.608 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:25:51.608 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:51.610 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:51.609+0000 7fde7629a640 10 librados: watch_flush enter 2024-03-25T18:25:51.610 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:51.609+0000 7fde7629a640 10 librados: watch_flush exit 2024-03-25T18:25:51.611 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:51.609+0000 7fde7629a640 20 client.4273.objecter shutdown clearing up homeless session... 2024-03-25T18:25:51.611 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:51.609+0000 7fde7629a640 10 client.4273.objecter successfully canceled tick 2024-03-25T18:25:51.611 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:51.609+0000 7fde7629a640 1 -- 172.21.15.155:0/2443924018 >> v1:172.21.15.167:6800/2919070503 conn(0x7fde5006dfb0 legacy=0x7fde50070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:51.611 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:51.609+0000 7fde7629a640 10 monclient: shutdown 2024-03-25T18:25:51.611 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:51.609+0000 7fde7629a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:51.611 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:51.609+0000 7fde7629a640 1 -- 172.21.15.155:0/2443924018 >> v1:172.21.15.167:6789/0 conn(0x7fde7013f710 legacy=0x7fde701dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:51.611 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:51.609+0000 7fde7629a640 1 -- 172.21.15.155:0/2443924018 shutdown_connections 2024-03-25T18:25:51.611 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:51.609+0000 7fde7629a640 1 -- 172.21.15.155:0/2443924018 >> 172.21.15.155:0/2443924018 conn(0x7fde700b7ea0 msgr2=0x7fde7014ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:51.611 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:51.609+0000 7fde7629a640 1 -- 172.21.15.155:0/2443924018 shutdown_connections 2024-03-25T18:25:51.611 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:51.609+0000 7fde7629a640 1 -- 172.21.15.155:0/2443924018 wait complete. 2024-03-25T18:25:51.611 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:51.609+0000 7fde7629a640 1 librados: shutdown 2024-03-25T18:25:51.622 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:58.429 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:58.545 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.544+0000 7f80d8d6d640 10 monclient: get_monmap_and_config 2024-03-25T18:25:58.545 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.544+0000 7f80d8d6d640 10 monclient: build_initial_monmap 2024-03-25T18:25:58.545 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.544+0000 7f80d8d6d640 10 monclient: monmap: 2024-03-25T18:25:58.546 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:25:58.546 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:25:58.546 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:25:58.545266+0000 2024-03-25T18:25:58.546 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:25:58.545266+0000 2024-03-25T18:25:58.546 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:58.546 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:25:58.546 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:25:58.546 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:25:58.546 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:25:58.546 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:58.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.544+0000 7f80d8d6d640 1 Processor -- start 2024-03-25T18:25:58.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.545+0000 7f80d8d6d640 1 -- start start 2024-03-25T18:25:58.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.545+0000 7f80d8d6d640 10 monclient: init 2024-03-25T18:25:58.547 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.545+0000 7f80d8d6d640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:58.548 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.545+0000 7f80d8d6d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:25:58.548 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.545+0000 7f80d8d6d640 10 monclient(hunting): picked mon.noname-a con 0x7f80d40a98f0 addr v1:172.21.15.155:6789/0 2024-03-25T18:25:58.548 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.545+0000 7f80d8d6d640 10 monclient(hunting): picked mon.noname-b con 0x7f80d413f710 addr v1:172.21.15.155:6790/0 2024-03-25T18:25:58.548 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.546+0000 7f80d8d6d640 10 monclient(hunting): picked mon.noname-c con 0x7f80d41430c0 addr v1:172.21.15.167:6789/0 2024-03-25T18:25:58.548 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.546+0000 7f80d8d6d640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80d4142520 con 0x7f80d40a98f0 2024-03-25T18:25:58.548 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.546+0000 7f80d8d6d640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80d40b28a0 con 0x7f80d41430c0 2024-03-25T18:25:58.548 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.546+0000 7f80d8d6d640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80d40b2a30 con 0x7f80d413f710 2024-03-25T18:25:58.548 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.546+0000 7f80d8d6d640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:58.548 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.546+0000 7f80d8d6d640 10 monclient(hunting): authenticate will time out at 7604.855999s 2024-03-25T18:25:58.550 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.549+0000 7f80d1d9b640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f80d413f710 0x7f80d41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:52490/0 (socket says 172.21.15.155:52490) 2024-03-25T18:25:58.550 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.549+0000 7f80d1d9b640 1 -- 172.21.15.155:0/1655660100 learned_addr learned my addr 172.21.15.155:0/1655660100 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:25:58.566 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.565+0000 7f80d159a640 1 -- 172.21.15.155:0/1655660100 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2807694014 0 0) 0x7f80d40b2a30 con 0x7f80d413f710 2024-03-25T18:25:58.566 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.565+0000 7f80d159a640 10 monclient(hunting): _negotiate 2024-03-25T18:25:58.566 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.565+0000 7f80d159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:58.566 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.565+0000 7f80d159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:58.566 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.565+0000 7f80d159a640 10 monclient(hunting): my global_id is 4277 2024-03-25T18:25:58.566 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.565+0000 7f80d159a640 1 -- 172.21.15.155:0/1655660100 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80bc003520 con 0x7f80d413f710 2024-03-25T18:25:58.574 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.573+0000 7f80d159a640 1 -- 172.21.15.155:0/1655660100 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1277380185 0 0) 0x7f80bc003520 con 0x7f80d413f710 2024-03-25T18:25:58.574 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.573+0000 7f80d159a640 1 -- 172.21.15.155:0/1655660100 >> v1:172.21.15.167:6789/0 conn(0x7f80d41430c0 legacy=0x7f80d414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:58.575 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.573+0000 7f80d159a640 1 -- 172.21.15.155:0/1655660100 >> v1:172.21.15.155:6789/0 conn(0x7f80d40a98f0 legacy=0x7f80d40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:58.575 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.573+0000 7f80d159a640 10 monclient: _finish_hunting 0 2024-03-25T18:25:58.575 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.574+0000 7f80d159a640 1 monclient: found mon.noname-b 2024-03-25T18:25:58.575 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.574+0000 7f80d159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:58.575 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.574+0000 7f80d159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-25T18:25:58.575 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.574+0000 7f80d159a640 1 -- 172.21.15.155:0/1655660100 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80d40b44c0 con 0x7f80d413f710 2024-03-25T18:25:58.575 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.574+0000 7f80d159a640 10 monclient: _finish_auth 0 2024-03-25T18:25:58.575 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.574+0000 7f80d159a640 10 monclient: _check_auth_tickets 2024-03-25T18:25:58.575 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.574+0000 7f80d159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:58.575 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.574+0000 7f80d159a640 1 -- 172.21.15.155:0/1655660100 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f80c8001670 con 0x7f80d413f710 2024-03-25T18:25:58.575 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.574+0000 7f80d159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:58.575 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.574+0000 7f80d159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:25:58.575 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.574+0000 7f80d159a640 10 monclient: dump: 2024-03-25T18:25:58.575 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:25:58.576 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:25:58.576 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:58.576 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:58.576 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:58.576 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:25:58.576 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:25:58.576 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:25:58.576 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:25:58.576 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:58.576 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.574+0000 7f80d8d6d640 5 monclient: authenticate success, global_id 4277 2024-03-25T18:25:58.576 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.574+0000 7f80d8d6d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:58.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.577+0000 7f80d159a640 1 -- 172.21.15.155:0/1655660100 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f80c80017d0 con 0x7f80d413f710 2024-03-25T18:25:58.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.577+0000 7f80d159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:58.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.577+0000 7f80d159a640 1 -- 172.21.15.155:0/1655660100 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f80c8001c20 con 0x7f80d413f710 2024-03-25T18:25:58.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.577+0000 7f80d8d6d640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:58.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.577+0000 7f80d8d6d640 4 set_mon_vals no callback set 2024-03-25T18:25:58.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.577+0000 7f80d159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:58.579 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.577+0000 7f80d159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:58.579 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.577+0000 7f80d159a640 10 monclient: dump: 2024-03-25T18:25:58.579 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:25:58.579 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:25:58.579 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:58.579 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:58.579 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:58.579 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:25:58.579 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:25:58.579 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:25:58.579 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:25:58.579 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:58.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.578+0000 7f80d8d6d640 10 monclient: shutdown 2024-03-25T18:25:58.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.578+0000 7f80d8d6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:58.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.578+0000 7f80d8d6d640 1 -- 172.21.15.155:0/1655660100 >> v1:172.21.15.155:6790/0 conn(0x7f80d413f710 legacy=0x7f80d41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:58.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.578+0000 7f80d8d6d640 1 -- 172.21.15.155:0/1655660100 shutdown_connections 2024-03-25T18:25:58.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.578+0000 7f80d8d6d640 1 -- 172.21.15.155:0/1655660100 >> 172.21.15.155:0/1655660100 conn(0x7f80d40b7ea0 msgr2=0x7f80d40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:58.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.578+0000 7f80d8d6d640 1 -- 172.21.15.155:0/1655660100 shutdown_connections 2024-03-25T18:25:58.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.578+0000 7f80d8d6d640 1 -- 172.21.15.155:0/1655660100 wait complete. 2024-03-25T18:25:58.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.579+0000 7f80d8d6d640 10 monclient: build_initial_monmap 2024-03-25T18:25:58.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.579+0000 7f80d8d6d640 10 monclient: monmap: 2024-03-25T18:25:58.582 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:25:58.582 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:58.582 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:25:58.582 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:25:58.582 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:58.582 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:25:58.582 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:25:58.582 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:25:58.582 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:25:58.583 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:58.583 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.579+0000 7f80d8d6d640 1 librados: starting msgr at 2024-03-25T18:25:58.583 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.579+0000 7f80d8d6d640 1 librados: starting objecter 2024-03-25T18:25:58.583 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.579+0000 7f80d8d6d640 1 Processor -- start 2024-03-25T18:25:58.583 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.579+0000 7f80d8d6d640 1 -- start start 2024-03-25T18:25:58.583 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.579+0000 7f80d8d6d640 1 librados: setting wanted keys 2024-03-25T18:25:58.583 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.579+0000 7f80d8d6d640 1 librados: calling monclient init 2024-03-25T18:25:58.583 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.579+0000 7f80d8d6d640 10 monclient: init 2024-03-25T18:25:58.583 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.580+0000 7f80d8d6d640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:58.583 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.580+0000 7f80d8d6d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:25:58.583 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.580+0000 7f80d8d6d640 10 monclient(hunting): picked mon.noname-c con 0x7f80d40a98f0 addr v1:172.21.15.155:6790/0 2024-03-25T18:25:58.583 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.580+0000 7f80d8d6d640 10 monclient(hunting): picked mon.noname-a con 0x7f80d413f710 addr v1:172.21.15.155:6789/0 2024-03-25T18:25:58.583 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.580+0000 7f80d8d6d640 10 monclient(hunting): picked mon.noname-b con 0x7f80d41430c0 addr v1:172.21.15.167:6789/0 2024-03-25T18:25:58.583 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.580+0000 7f80d8d6d640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80d41db960 con 0x7f80d413f710 2024-03-25T18:25:58.583 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.580+0000 7f80d8d6d640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80d41db560 con 0x7f80d41430c0 2024-03-25T18:25:58.584 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.580+0000 7f80d8d6d640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80d40b15f0 con 0x7f80d40a98f0 2024-03-25T18:25:58.584 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.580+0000 7f80d8d6d640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:58.584 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.580+0000 7f80d8d6d640 10 monclient(hunting): authenticate will time out at 7604.890282s 2024-03-25T18:25:58.589 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.588+0000 7f80d2d9d640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f80d41430c0 0x7f80d41d42c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:33470/0 (socket says 172.21.15.155:33470) 2024-03-25T18:25:58.589 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.588+0000 7f80d2d9d640 1 -- 172.21.15.155:0/1562265200 learned_addr learned my addr 172.21.15.155:0/1562265200 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:25:58.592 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.591+0000 7f80c2ffd640 10 client.?.objecter ms_handle_connect 0x7f80d41430c0 2024-03-25T18:25:58.592 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.591+0000 7f80c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:58.602 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.601+0000 7f80c2ffd640 10 client.?.objecter ms_handle_connect 0x7f80d40a98f0 2024-03-25T18:25:58.602 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.601+0000 7f80c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:58.602 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.601+0000 7f80c2ffd640 1 -- 172.21.15.155:0/1562265200 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1694051531 0 0) 0x7f80d41db560 con 0x7f80d41430c0 2024-03-25T18:25:58.602 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.601+0000 7f80c2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:58.602 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.601+0000 7f80c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:58.602 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.601+0000 7f80c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:58.603 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.601+0000 7f80c2ffd640 10 monclient(hunting): my global_id is 4279 2024-03-25T18:25:58.603 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.601+0000 7f80c2ffd640 1 -- 172.21.15.155:0/1562265200 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80b4004eb0 con 0x7f80d41430c0 2024-03-25T18:25:58.608 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.607+0000 7f80c2ffd640 10 client.?.objecter ms_handle_connect 0x7f80d413f710 2024-03-25T18:25:58.608 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.607+0000 7f80c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:58.610 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.609+0000 7f80c2ffd640 1 -- 172.21.15.155:0/1562265200 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3824535918 0 0) 0x7f80d40b15f0 con 0x7f80d40a98f0 2024-03-25T18:25:58.610 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.609+0000 7f80c2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:58.610 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.609+0000 7f80c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:58.610 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.609+0000 7f80c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:58.611 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.609+0000 7f80c2ffd640 10 monclient(hunting): my global_id is 4280 2024-03-25T18:25:58.611 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.609+0000 7f80c2ffd640 1 -- 172.21.15.155:0/1562265200 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80d41db560 con 0x7f80d40a98f0 2024-03-25T18:25:58.619 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.619+0000 7f80c2ffd640 1 -- 172.21.15.155:0/1562265200 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 588007641 0 0) 0x7f80d41db960 con 0x7f80d413f710 2024-03-25T18:25:58.620 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.619+0000 7f80c2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:58.620 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.619+0000 7f80c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:58.620 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.619+0000 7f80c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:58.620 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.619+0000 7f80c2ffd640 10 monclient(hunting): my global_id is 4278 2024-03-25T18:25:58.620 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.619+0000 7f80c2ffd640 1 -- 172.21.15.155:0/1562265200 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80d40b15f0 con 0x7f80d413f710 2024-03-25T18:25:58.623 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.622+0000 7f80c2ffd640 1 -- 172.21.15.155:0/1562265200 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3341030470 0 0) 0x7f80d41db560 con 0x7f80d40a98f0 2024-03-25T18:25:58.623 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.622+0000 7f80c2ffd640 1 -- 172.21.15.155:0/1562265200 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f80d41db960 con 0x7f80d40a98f0 2024-03-25T18:25:58.623 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.622+0000 7f80c2ffd640 1 -- 172.21.15.155:0/1562265200 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f80c40018b0 con 0x7f80d40a98f0 2024-03-25T18:25:58.623 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.622+0000 7f80c2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:58.623 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.622+0000 7f80c2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:58.623 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.622+0000 7f80c2ffd640 10 monclient(hunting): dump: 2024-03-25T18:25:58.623 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:25:58.623 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:25:58.623 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:58.623 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:58.623 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:58.624 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:25:58.624 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:25:58.624 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:25:58.624 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:25:58.624 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:58.624 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.622+0000 7f80c2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:58.633 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.632+0000 7f80c2ffd640 1 -- 172.21.15.155:0/1562265200 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 421039611 0 0) 0x7f80b4004eb0 con 0x7f80d41430c0 2024-03-25T18:25:58.633 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.632+0000 7f80c2ffd640 1 -- 172.21.15.155:0/1562265200 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f80d41db560 con 0x7f80d41430c0 2024-03-25T18:25:58.633 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.632+0000 7f80c2ffd640 1 -- 172.21.15.155:0/1562265200 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f80cc001c30 con 0x7f80d41430c0 2024-03-25T18:25:58.633 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.632+0000 7f80c2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:58.633 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.632+0000 7f80c2ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:58.633 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.632+0000 7f80c2ffd640 10 monclient(hunting): dump: 2024-03-25T18:25:58.633 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:25:58.633 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:25:58.633 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:58.633 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:58.634 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:58.634 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:25:58.634 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:25:58.634 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:25:58.634 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:25:58.634 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:58.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.634+0000 7f80c2ffd640 1 -- 172.21.15.155:0/1562265200 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4087399519 0 0) 0x7f80d41db960 con 0x7f80d40a98f0 2024-03-25T18:25:58.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.634+0000 7f80c2ffd640 1 -- 172.21.15.155:0/1562265200 >> v1:172.21.15.167:6789/0 conn(0x7f80d41430c0 legacy=0x7f80d41d42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:58.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.634+0000 7f80c2ffd640 1 -- 172.21.15.155:0/1562265200 >> v1:172.21.15.155:6789/0 conn(0x7f80d413f710 legacy=0x7f80d414d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:58.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.634+0000 7f80c2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:25:58.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.634+0000 7f80c2ffd640 1 monclient: found mon.c 2024-03-25T18:25:58.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.634+0000 7f80c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:58.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.634+0000 7f80c2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:25:58.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.634+0000 7f80c2ffd640 1 -- 172.21.15.155:0/1562265200 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80d40b1780 con 0x7f80d40a98f0 2024-03-25T18:25:58.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.634+0000 7f80c2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:25:58.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.634+0000 7f80c2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:25:58.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.634+0000 7f80c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:58.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.634+0000 7f80d8d6d640 5 monclient: authenticate success, global_id 4280 2024-03-25T18:25:58.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.634+0000 7f80d8d6d640 10 monclient: _renew_subs 2024-03-25T18:25:58.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.634+0000 7f80d8d6d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:25:58.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.634+0000 7f80d8d6d640 1 -- 172.21.15.155:0/1562265200 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80d414ead0 con 0x7f80d40a98f0 2024-03-25T18:25:58.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.634+0000 7f80d8d6d640 10 client.4280.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:58.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.634+0000 7f80d8d6d640 10 monclient: _renew_subs 2024-03-25T18:25:58.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.634+0000 7f80d8d6d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:25:58.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.634+0000 7f80d8d6d640 1 -- 172.21.15.155:0/1562265200 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f80d414e270 con 0x7f80d40a98f0 2024-03-25T18:25:58.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.635+0000 7f80d8d6d640 1 librados: init done 2024-03-25T18:25:58.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.635+0000 7f80d8d6d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:58.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.636+0000 7f80d8d6d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:58.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.636+0000 7f80d8d6d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:25:58.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.636+0000 7f80d8d6d640 1 -- 172.21.15.155:0/1562265200 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80d414ee20 con 0x7f80d40a98f0 2024-03-25T18:25:58.946 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.945+0000 7f80c2ffd640 1 -- 172.21.15.155:0/1562265200 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f80c4003610 con 0x7f80d40a98f0 2024-03-25T18:25:58.946 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.945+0000 7f80c2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:58.946 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.945+0000 7f80c2ffd640 1 -- 172.21.15.155:0/1562265200 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f80c4003a60 con 0x7f80d40a98f0 2024-03-25T18:25:58.946 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.945+0000 7f80c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:58.946 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.945+0000 7f80c2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:58.946 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.945+0000 7f80c2ffd640 10 monclient: dump: 2024-03-25T18:25:58.946 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:25:58.946 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:25:58.946 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:58.946 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:25:58.946 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:58.946 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:25:58.947 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:25:58.947 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:25:58.947 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:25:58.947 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:25:58.947 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.945+0000 7f80d0d99640 4 set_mon_vals no callback set 2024-03-25T18:25:58.947 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.946+0000 7f80c2ffd640 1 -- 172.21.15.155:0/1562265200 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7f80c4003c80 con 0x7f80d40a98f0 2024-03-25T18:25:58.948 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.946+0000 7f80d1d9b640 10 monclient: get_auth_request con 0x7f80b406df70 auth_method 0 2024-03-25T18:25:58.948 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.946+0000 7f80c2ffd640 10 client.4280.objecter ms_handle_connect 0x7f80b406df70 2024-03-25T18:25:58.954 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.953+0000 7f80c2ffd640 1 -- 172.21.15.155:0/1562265200 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7f80c40535f0 con 0x7f80d40a98f0 2024-03-25T18:25:58.954 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.953+0000 7f80c2ffd640 10 client.4280.objecter ms_dispatch 0x7f80d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:58.955 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.953+0000 7f80c2ffd640 3 client.4280.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:58.955 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.953+0000 7f80c2ffd640 3 client.4280.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:58.955 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.953+0000 7f80c2ffd640 20 client.4280.objecter dump_active .. 0 homeless 2024-03-25T18:25:58.955 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.953+0000 7f80c2ffd640 1 -- 172.21.15.155:0/1562265200 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f80c404f100 con 0x7f80d40a98f0 2024-03-25T18:25:58.955 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.953+0000 7f80c2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:58.955 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:58.953+0000 7f80c2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:59.182 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:59.181+0000 7f80d8d6d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:59.182 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:59.181+0000 7f80d8d6d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:59.182 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:59.181+0000 7f80d8d6d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:25:59.182 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:59.181+0000 7f80d8d6d640 1 -- 172.21.15.155:0/1562265200 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f80d40627b0 con 0x7f80d40a98f0 2024-03-25T18:25:59.291 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:59.290+0000 7f80c2ffd640 1 -- 172.21.15.155:0/1562265200 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7f80c4051030 con 0x7f80d40a98f0 2024-03-25T18:25:59.292 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:59.290+0000 7f80c2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:59.292 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:59.290+0000 7f80c2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:59.292 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:25:59.292 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:59.295 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:59.293+0000 7f80d8d6d640 10 librados: watch_flush enter 2024-03-25T18:25:59.295 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:59.293+0000 7f80d8d6d640 10 librados: watch_flush exit 2024-03-25T18:25:59.295 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:59.293+0000 7f80d8d6d640 20 client.4280.objecter shutdown clearing up homeless session... 2024-03-25T18:25:59.295 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:59.293+0000 7f80d8d6d640 10 client.4280.objecter successfully canceled tick 2024-03-25T18:25:59.295 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:59.293+0000 7f80d8d6d640 1 -- 172.21.15.155:0/1562265200 >> v1:172.21.15.167:6800/2919070503 conn(0x7f80b406df70 legacy=0x7f80b4070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:59.295 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:59.293+0000 7f80d8d6d640 10 monclient: shutdown 2024-03-25T18:25:59.296 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:59.293+0000 7f80d8d6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:59.296 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:59.293+0000 7f80d8d6d640 1 -- 172.21.15.155:0/1562265200 >> v1:172.21.15.155:6790/0 conn(0x7f80d40a98f0 legacy=0x7f80d41dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:59.296 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:59.293+0000 7f80d8d6d640 1 -- 172.21.15.155:0/1562265200 shutdown_connections 2024-03-25T18:25:59.296 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:59.293+0000 7f80d8d6d640 1 -- 172.21.15.155:0/1562265200 >> 172.21.15.155:0/1562265200 conn(0x7f80d40b7ea0 msgr2=0x7f80d4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:59.296 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:59.293+0000 7f80d8d6d640 1 -- 172.21.15.155:0/1562265200 shutdown_connections 2024-03-25T18:25:59.296 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:59.293+0000 7f80d8d6d640 1 -- 172.21.15.155:0/1562265200 wait complete. 2024-03-25T18:25:59.296 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:25:59.294+0000 7f80d8d6d640 1 librados: shutdown 2024-03-25T18:25:59.306 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:00.484 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:01:34.088 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:00.495 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:01:34.100 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:00.549 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:01:34.127 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:00.549 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:01:34.154 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:00.553 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:01:34.157 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:00.562 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:01:34.141 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:00.564 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:01:34.169 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:00.619 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:01:34.223 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:00.619 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:01:34.197 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:06.112 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:06.232 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.230+0000 7fa6e3e5c640 10 monclient: get_monmap_and_config 2024-03-25T18:26:06.232 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.230+0000 7fa6e3e5c640 10 monclient: build_initial_monmap 2024-03-25T18:26:06.232 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.230+0000 7fa6e3e5c640 10 monclient: monmap: 2024-03-25T18:26:06.232 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:26:06.232 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:26:06.232 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:26:06.231646+0000 2024-03-25T18:26:06.232 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:26:06.231646+0000 2024-03-25T18:26:06.232 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:06.232 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:26:06.232 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:26:06.232 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:26:06.232 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:26:06.233 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:26:06.233 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.231+0000 7fa6e3e5c640 1 Processor -- start 2024-03-25T18:26:06.234 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.231+0000 7fa6e3e5c640 1 -- start start 2024-03-25T18:26:06.234 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.231+0000 7fa6e3e5c640 10 monclient: init 2024-03-25T18:26:06.234 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.232+0000 7fa6e3e5c640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:06.234 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.232+0000 7fa6e3e5c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:26:06.234 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.232+0000 7fa6e3e5c640 10 monclient(hunting): picked mon.noname-a con 0x7fa6dc0a98f0 addr v1:172.21.15.155:6789/0 2024-03-25T18:26:06.234 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.232+0000 7fa6e3e5c640 10 monclient(hunting): picked mon.noname-b con 0x7fa6dc13f710 addr v1:172.21.15.155:6790/0 2024-03-25T18:26:06.234 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.232+0000 7fa6e3e5c640 10 monclient(hunting): picked mon.noname-c con 0x7fa6dc1430c0 addr v1:172.21.15.167:6789/0 2024-03-25T18:26:06.234 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.232+0000 7fa6e3e5c640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6dc142520 con 0x7fa6dc0a98f0 2024-03-25T18:26:06.234 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.232+0000 7fa6e3e5c640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6dc0b28a0 con 0x7fa6dc1430c0 2024-03-25T18:26:06.234 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.232+0000 7fa6e3e5c640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6dc0b2a30 con 0x7fa6dc13f710 2024-03-25T18:26:06.235 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.232+0000 7fa6e3e5c640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:06.235 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.232+0000 7fa6e3e5c640 10 monclient(hunting): authenticate will time out at 7612.542682s 2024-03-25T18:26:06.246 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.245+0000 7fa6e1bf8640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fa6dc0a98f0 0x7fa6dc0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:52432/0 (socket says 172.21.15.155:52432) 2024-03-25T18:26:06.246 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.245+0000 7fa6e1bf8640 1 -- 172.21.15.155:0/1068978604 learned_addr learned my addr 172.21.15.155:0/1068978604 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:26:06.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.260+0000 7fa6e0bf6640 1 -- 172.21.15.155:0/1068978604 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2216923785 0 0) 0x7fa6dc142520 con 0x7fa6dc0a98f0 2024-03-25T18:26:06.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.260+0000 7fa6e0bf6640 10 monclient(hunting): _negotiate 2024-03-25T18:26:06.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.260+0000 7fa6e0bf6640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:06.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.260+0000 7fa6e0bf6640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:06.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.260+0000 7fa6e0bf6640 10 monclient(hunting): my global_id is 4281 2024-03-25T18:26:06.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.260+0000 7fa6e0bf6640 1 -- 172.21.15.155:0/1068978604 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6c8003520 con 0x7fa6dc0a98f0 2024-03-25T18:26:06.266 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.265+0000 7fa6e0bf6640 1 -- 172.21.15.155:0/1068978604 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 275898934 0 0) 0x7fa6dc0b2a30 con 0x7fa6dc13f710 2024-03-25T18:26:06.266 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.265+0000 7fa6e0bf6640 10 monclient(hunting): _negotiate 2024-03-25T18:26:06.266 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.265+0000 7fa6e0bf6640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:06.266 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.265+0000 7fa6e0bf6640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:06.266 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.265+0000 7fa6e0bf6640 10 monclient(hunting): my global_id is 4283 2024-03-25T18:26:06.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.265+0000 7fa6e0bf6640 1 -- 172.21.15.155:0/1068978604 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6dc142520 con 0x7fa6dc13f710 2024-03-25T18:26:06.269 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.268+0000 7fa6e0bf6640 1 -- 172.21.15.155:0/1068978604 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 954609991 0 0) 0x7fa6c8003520 con 0x7fa6dc0a98f0 2024-03-25T18:26:06.269 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.268+0000 7fa6e0bf6640 1 -- 172.21.15.155:0/1068978604 >> v1:172.21.15.155:6790/0 conn(0x7fa6dc13f710 legacy=0x7fa6dc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:06.269 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.268+0000 7fa6e0bf6640 1 -- 172.21.15.155:0/1068978604 >> v1:172.21.15.167:6789/0 conn(0x7fa6dc1430c0 legacy=0x7fa6dc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:06.269 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.268+0000 7fa6e0bf6640 10 monclient: _finish_hunting 0 2024-03-25T18:26:06.269 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.268+0000 7fa6e0bf6640 1 monclient: found mon.noname-a 2024-03-25T18:26:06.270 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.268+0000 7fa6e0bf6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:06.270 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.268+0000 7fa6e0bf6640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-25T18:26:06.270 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.268+0000 7fa6e0bf6640 1 -- 172.21.15.155:0/1068978604 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6dc0b44c0 con 0x7fa6dc0a98f0 2024-03-25T18:26:06.270 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.268+0000 7fa6e0bf6640 10 monclient: _finish_auth 0 2024-03-25T18:26:06.270 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.268+0000 7fa6e0bf6640 10 monclient: _check_auth_tickets 2024-03-25T18:26:06.270 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.268+0000 7fa6e0bf6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:06.270 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.268+0000 7fa6e3e5c640 5 monclient: authenticate success, global_id 4281 2024-03-25T18:26:06.270 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.268+0000 7fa6e3e5c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:06.271 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.269+0000 7fa6e0bf6640 1 -- 172.21.15.155:0/1068978604 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fa6d8001650 con 0x7fa6dc0a98f0 2024-03-25T18:26:06.271 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.269+0000 7fa6e0bf6640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:06.271 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.269+0000 7fa6e0bf6640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:26:06.271 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.269+0000 7fa6e0bf6640 10 monclient: dump: 2024-03-25T18:26:06.271 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:26:06.271 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:26:06.271 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:06.272 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:06.272 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:06.272 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:26:06.272 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:26:06.272 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:26:06.272 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:26:06.272 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:26:06.272 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.269+0000 7fa6e3e5c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:06.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.282+0000 7fa6e0bf6640 1 -- 172.21.15.155:0/1068978604 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa6d80018a0 con 0x7fa6dc0a98f0 2024-03-25T18:26:06.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.282+0000 7fa6e0bf6640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:06.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.282+0000 7fa6e0bf6640 1 -- 172.21.15.155:0/1068978604 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fa6d8001c30 con 0x7fa6dc0a98f0 2024-03-25T18:26:06.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.282+0000 7fa6e3e5c640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:06.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.282+0000 7fa6e3e5c640 4 set_mon_vals no callback set 2024-03-25T18:26:06.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.282+0000 7fa6e0bf6640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:06.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.282+0000 7fa6e0bf6640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:06.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.282+0000 7fa6e0bf6640 10 monclient: dump: 2024-03-25T18:26:06.283 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:26:06.283 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:26:06.283 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:06.284 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:06.284 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:06.284 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:26:06.284 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:26:06.284 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:26:06.284 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:26:06.284 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:26:06.285 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.283+0000 7fa6e3e5c640 10 monclient: shutdown 2024-03-25T18:26:06.285 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.283+0000 7fa6e3e5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:06.285 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.283+0000 7fa6e3e5c640 1 -- 172.21.15.155:0/1068978604 >> v1:172.21.15.155:6789/0 conn(0x7fa6dc0a98f0 legacy=0x7fa6dc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:06.286 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.283+0000 7fa6e3e5c640 1 -- 172.21.15.155:0/1068978604 shutdown_connections 2024-03-25T18:26:06.286 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.283+0000 7fa6e3e5c640 1 -- 172.21.15.155:0/1068978604 >> 172.21.15.155:0/1068978604 conn(0x7fa6dc0b7ea0 msgr2=0x7fa6dc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:06.286 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.283+0000 7fa6e3e5c640 1 -- 172.21.15.155:0/1068978604 shutdown_connections 2024-03-25T18:26:06.286 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.283+0000 7fa6e3e5c640 1 -- 172.21.15.155:0/1068978604 wait complete. 2024-03-25T18:26:06.286 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.284+0000 7fa6e3e5c640 10 monclient: build_initial_monmap 2024-03-25T18:26:06.286 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.284+0000 7fa6e3e5c640 10 monclient: monmap: 2024-03-25T18:26:06.286 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:26:06.286 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:06.286 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:26:06.286 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:26:06.286 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:06.286 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:26:06.286 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:26:06.287 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:26:06.287 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:26:06.287 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:26:06.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.284+0000 7fa6e3e5c640 1 librados: starting msgr at 2024-03-25T18:26:06.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.284+0000 7fa6e3e5c640 1 librados: starting objecter 2024-03-25T18:26:06.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.284+0000 7fa6e3e5c640 1 Processor -- start 2024-03-25T18:26:06.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.284+0000 7fa6e3e5c640 1 -- start start 2024-03-25T18:26:06.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.284+0000 7fa6e3e5c640 1 librados: setting wanted keys 2024-03-25T18:26:06.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.284+0000 7fa6e3e5c640 1 librados: calling monclient init 2024-03-25T18:26:06.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.284+0000 7fa6e3e5c640 10 monclient: init 2024-03-25T18:26:06.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.285+0000 7fa6e3e5c640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:06.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.285+0000 7fa6e3e5c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:26:06.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.285+0000 7fa6e3e5c640 10 monclient(hunting): picked mon.noname-b con 0x7fa6dc0a98f0 addr v1:172.21.15.167:6789/0 2024-03-25T18:26:06.289 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.285+0000 7fa6e3e5c640 10 monclient(hunting): picked mon.noname-a con 0x7fa6dc13f710 addr v1:172.21.15.155:6789/0 2024-03-25T18:26:06.289 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.285+0000 7fa6e3e5c640 10 monclient(hunting): picked mon.noname-c con 0x7fa6dc1430c0 addr v1:172.21.15.155:6790/0 2024-03-25T18:26:06.289 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.285+0000 7fa6e3e5c640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6dc1db110 con 0x7fa6dc13f710 2024-03-25T18:26:06.289 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.285+0000 7fa6e3e5c640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6dc138d90 con 0x7fa6dc0a98f0 2024-03-25T18:26:06.289 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.285+0000 7fa6e3e5c640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6dc1376a0 con 0x7fa6dc1430c0 2024-03-25T18:26:06.289 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.285+0000 7fa6e3e5c640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:06.289 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.285+0000 7fa6e3e5c640 10 monclient(hunting): authenticate will time out at 7612.595707s 2024-03-25T18:26:06.290 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.290+0000 7fa6e23f9640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fa6dc1430c0 0x7fa6dc132f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:57430/0 (socket says 172.21.15.155:57430) 2024-03-25T18:26:06.291 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.290+0000 7fa6e23f9640 1 -- 172.21.15.155:0/2016409006 learned_addr learned my addr 172.21.15.155:0/2016409006 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:26:06.292 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.291+0000 7fa6d27fc640 10 client.?.objecter ms_handle_connect 0x7fa6dc1430c0 2024-03-25T18:26:06.292 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.291+0000 7fa6d27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:06.300 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.299+0000 7fa6d27fc640 1 -- 172.21.15.155:0/2016409006 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 561664709 0 0) 0x7fa6dc1376a0 con 0x7fa6dc1430c0 2024-03-25T18:26:06.300 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.299+0000 7fa6d27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:26:06.300 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.299+0000 7fa6d27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:06.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.299+0000 7fa6d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:06.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.299+0000 7fa6d27fc640 10 monclient(hunting): my global_id is 4286 2024-03-25T18:26:06.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.299+0000 7fa6d27fc640 1 -- 172.21.15.155:0/2016409006 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6c0004eb0 con 0x7fa6dc1430c0 2024-03-25T18:26:06.313 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.313+0000 7fa6d27fc640 10 client.?.objecter ms_handle_connect 0x7fa6dc13f710 2024-03-25T18:26:06.313 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.313+0000 7fa6d27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:06.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.314+0000 7fa6d27fc640 1 -- 172.21.15.155:0/2016409006 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3211583804 0 0) 0x7fa6c0004eb0 con 0x7fa6dc1430c0 2024-03-25T18:26:06.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.315+0000 7fa6d27fc640 1 -- 172.21.15.155:0/2016409006 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa6dc1376a0 con 0x7fa6dc1430c0 2024-03-25T18:26:06.316 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.315+0000 7fa6d27fc640 1 -- 172.21.15.155:0/2016409006 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fa6d4001c30 con 0x7fa6dc1430c0 2024-03-25T18:26:06.316 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.315+0000 7fa6d27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:06.316 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.315+0000 7fa6d27fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:06.316 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.315+0000 7fa6d27fc640 10 monclient(hunting): dump: 2024-03-25T18:26:06.316 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:26:06.316 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:26:06.316 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:06.316 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:06.316 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:06.316 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:26:06.316 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:26:06.316 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:26:06.316 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:26:06.317 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:26:06.317 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.315+0000 7fa6d27fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:06.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.317+0000 7fa6d27fc640 10 client.?.objecter ms_handle_connect 0x7fa6dc0a98f0 2024-03-25T18:26:06.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.317+0000 7fa6d27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:06.327 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.325+0000 7fa6d27fc640 1 -- 172.21.15.155:0/2016409006 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2062434324 0 0) 0x7fa6dc1376a0 con 0x7fa6dc1430c0 2024-03-25T18:26:06.327 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.325+0000 7fa6d27fc640 1 -- 172.21.15.155:0/2016409006 >> v1:172.21.15.167:6789/0 conn(0x7fa6dc0a98f0 legacy=0x7fa6dc1369a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:06.328 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.325+0000 7fa6d27fc640 1 -- 172.21.15.155:0/2016409006 >> v1:172.21.15.155:6789/0 conn(0x7fa6dc13f710 legacy=0x7fa6dc1daa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:06.328 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.325+0000 7fa6d27fc640 10 monclient: _finish_hunting 0 2024-03-25T18:26:06.328 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.325+0000 7fa6d27fc640 1 monclient: found mon.c 2024-03-25T18:26:06.328 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.325+0000 7fa6d27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:06.328 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.325+0000 7fa6d27fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:26:06.328 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.325+0000 7fa6d27fc640 1 -- 172.21.15.155:0/2016409006 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6dc0b7260 con 0x7fa6dc1430c0 2024-03-25T18:26:06.328 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.325+0000 7fa6d27fc640 10 monclient: _finish_auth 0 2024-03-25T18:26:06.328 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.325+0000 7fa6d27fc640 10 monclient: _check_auth_tickets 2024-03-25T18:26:06.328 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.325+0000 7fa6d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:06.328 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.325+0000 7fa6e3e5c640 5 monclient: authenticate success, global_id 4286 2024-03-25T18:26:06.328 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.326+0000 7fa6e3e5c640 10 monclient: _renew_subs 2024-03-25T18:26:06.328 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.326+0000 7fa6e3e5c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:26:06.328 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.326+0000 7fa6e3e5c640 1 -- 172.21.15.155:0/2016409006 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6dc0b74b0 con 0x7fa6dc1430c0 2024-03-25T18:26:06.329 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.326+0000 7fa6e3e5c640 10 client.4286.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:06.329 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.326+0000 7fa6e3e5c640 10 monclient: _renew_subs 2024-03-25T18:26:06.329 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.326+0000 7fa6e3e5c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:26:06.329 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.326+0000 7fa6e3e5c640 1 -- 172.21.15.155:0/2016409006 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6dc1337f0 con 0x7fa6dc1430c0 2024-03-25T18:26:06.329 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.326+0000 7fa6e3e5c640 1 librados: init done 2024-03-25T18:26:06.329 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.327+0000 7fa6e3e5c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:06.329 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.327+0000 7fa6e3e5c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:06.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.327+0000 7fa6e3e5c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:26:06.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.327+0000 7fa6e3e5c640 1 -- 172.21.15.155:0/2016409006 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6dc133a90 con 0x7fa6dc1430c0 2024-03-25T18:26:06.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.638+0000 7fa6d27fc640 1 -- 172.21.15.155:0/2016409006 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa6d4003970 con 0x7fa6dc1430c0 2024-03-25T18:26:06.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.638+0000 7fa6d27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:06.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.638+0000 7fa6d27fc640 1 -- 172.21.15.155:0/2016409006 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fa6d4003be0 con 0x7fa6dc1430c0 2024-03-25T18:26:06.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.638+0000 7fa6d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:06.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.638+0000 7fa6d27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:06.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.638+0000 7fa6d27fc640 10 monclient: dump: 2024-03-25T18:26:06.639 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:26:06.639 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:26:06.639 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:06.639 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:06.639 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:06.640 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:26:06.640 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:26:06.640 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:26:06.640 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:26:06.640 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:26:06.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.638+0000 7fa6d3fff640 4 set_mon_vals no callback set 2024-03-25T18:26:06.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.639+0000 7fa6d27fc640 1 -- 172.21.15.155:0/2016409006 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7fa6d4003e00 con 0x7fa6dc1430c0 2024-03-25T18:26:06.641 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.639+0000 7fa6e1bf8640 10 monclient: get_auth_request con 0x7fa6c006dc70 auth_method 0 2024-03-25T18:26:06.641 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.639+0000 7fa6d27fc640 10 client.4286.objecter ms_handle_connect 0x7fa6c006dc70 2024-03-25T18:26:06.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.644+0000 7fa6d27fc640 1 -- 172.21.15.155:0/2016409006 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7fa6d4004800 con 0x7fa6dc1430c0 2024-03-25T18:26:06.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.644+0000 7fa6d27fc640 10 client.4286.objecter ms_dispatch 0x7fa6dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:06.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.644+0000 7fa6d27fc640 3 client.4286.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:06.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.644+0000 7fa6d27fc640 3 client.4286.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:06.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.644+0000 7fa6d27fc640 20 client.4286.objecter dump_active .. 0 homeless 2024-03-25T18:26:06.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.644+0000 7fa6d27fc640 1 -- 172.21.15.155:0/2016409006 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa6d4050420 con 0x7fa6dc1430c0 2024-03-25T18:26:06.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.645+0000 7fa6d27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:06.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.645+0000 7fa6d27fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:06.896 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.895+0000 7fa6e3e5c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:06.896 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.895+0000 7fa6e3e5c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:06.896 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.895+0000 7fa6e3e5c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:26:06.896 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.895+0000 7fa6e3e5c640 1 -- 172.21.15.155:0/2016409006 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa6dc137830 con 0x7fa6dc1430c0 2024-03-25T18:26:06.938 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.937+0000 7fa6d27fc640 1 -- 172.21.15.155:0/2016409006 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7fa6d4052350 con 0x7fa6dc1430c0 2024-03-25T18:26:06.938 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.937+0000 7fa6d27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:06.938 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.937+0000 7fa6d27fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:06.939 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:26:06.939 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:06.941 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.939+0000 7fa6e3e5c640 10 librados: watch_flush enter 2024-03-25T18:26:06.941 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.939+0000 7fa6e3e5c640 10 librados: watch_flush exit 2024-03-25T18:26:06.941 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.939+0000 7fa6e3e5c640 20 client.4286.objecter shutdown clearing up homeless session... 2024-03-25T18:26:06.941 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.939+0000 7fa6e3e5c640 10 client.4286.objecter successfully canceled tick 2024-03-25T18:26:06.941 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.940+0000 7fa6e3e5c640 1 -- 172.21.15.155:0/2016409006 >> v1:172.21.15.167:6800/2919070503 conn(0x7fa6c006dc70 legacy=0x7fa6c0070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:06.941 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.940+0000 7fa6e3e5c640 10 monclient: shutdown 2024-03-25T18:26:06.941 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.940+0000 7fa6e3e5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:06.941 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.940+0000 7fa6e3e5c640 1 -- 172.21.15.155:0/2016409006 >> v1:172.21.15.155:6790/0 conn(0x7fa6dc1430c0 legacy=0x7fa6dc132f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:06.942 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.940+0000 7fa6e3e5c640 1 -- 172.21.15.155:0/2016409006 shutdown_connections 2024-03-25T18:26:06.942 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.940+0000 7fa6e3e5c640 1 -- 172.21.15.155:0/2016409006 >> 172.21.15.155:0/2016409006 conn(0x7fa6dc0b7ea0 msgr2=0x7fa6dc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:06.942 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.940+0000 7fa6e3e5c640 1 -- 172.21.15.155:0/2016409006 shutdown_connections 2024-03-25T18:26:06.942 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.940+0000 7fa6e3e5c640 1 -- 172.21.15.155:0/2016409006 wait complete. 2024-03-25T18:26:06.942 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:06.940+0000 7fa6e3e5c640 1 librados: shutdown 2024-03-25T18:26:06.952 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:13.760 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:13.877 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.876+0000 7f6e5a864640 10 monclient: get_monmap_and_config 2024-03-25T18:26:13.878 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.876+0000 7f6e5a864640 10 monclient: build_initial_monmap 2024-03-25T18:26:13.878 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.876+0000 7f6e5a864640 10 monclient: monmap: 2024-03-25T18:26:13.878 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:26:13.878 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:26:13.878 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:26:13.877908+0000 2024-03-25T18:26:13.878 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:26:13.877908+0000 2024-03-25T18:26:13.878 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:13.878 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:26:13.878 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:26:13.878 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:26:13.878 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:26:13.879 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:26:13.880 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.877+0000 7f6e5a864640 1 Processor -- start 2024-03-25T18:26:13.880 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.877+0000 7f6e5a864640 1 -- start start 2024-03-25T18:26:13.880 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.877+0000 7f6e5a864640 10 monclient: init 2024-03-25T18:26:13.880 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.878+0000 7f6e5a864640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:13.880 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.878+0000 7f6e5a864640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:26:13.880 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.878+0000 7f6e5a864640 10 monclient(hunting): picked mon.noname-a con 0x7f6e540a98f0 addr v1:172.21.15.155:6789/0 2024-03-25T18:26:13.880 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.878+0000 7f6e5a864640 10 monclient(hunting): picked mon.noname-b con 0x7f6e5413f710 addr v1:172.21.15.155:6790/0 2024-03-25T18:26:13.880 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.878+0000 7f6e5a864640 10 monclient(hunting): picked mon.noname-c con 0x7f6e541430c0 addr v1:172.21.15.167:6789/0 2024-03-25T18:26:13.880 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.878+0000 7f6e5a864640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e54142520 con 0x7f6e540a98f0 2024-03-25T18:26:13.880 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.878+0000 7f6e5a864640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e540b28a0 con 0x7f6e541430c0 2024-03-25T18:26:13.880 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.878+0000 7f6e5a864640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e540b2a30 con 0x7f6e5413f710 2024-03-25T18:26:13.880 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.878+0000 7f6e5a864640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:13.881 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.878+0000 7f6e5a864640 10 monclient(hunting): authenticate will time out at 7620.188623s 2024-03-25T18:26:13.882 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.881+0000 7f6e537fe640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f6e5413f710 0x7f6e541429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:39352/0 (socket says 172.21.15.155:39352) 2024-03-25T18:26:13.882 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.882+0000 7f6e537fe640 1 -- 172.21.15.155:0/3351573062 learned_addr learned my addr 172.21.15.155:0/3351573062 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:26:13.901 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.900+0000 7f6e52ffd640 1 -- 172.21.15.155:0/3351573062 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3935036844 0 0) 0x7f6e540b28a0 con 0x7f6e541430c0 2024-03-25T18:26:13.901 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.900+0000 7f6e52ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:13.901 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.900+0000 7f6e52ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:13.901 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.900+0000 7f6e52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:13.901 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.900+0000 7f6e52ffd640 10 monclient(hunting): my global_id is 4288 2024-03-25T18:26:13.901 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.900+0000 7f6e52ffd640 1 -- 172.21.15.155:0/3351573062 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e3c003520 con 0x7f6e541430c0 2024-03-25T18:26:13.912 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.911+0000 7f6e52ffd640 1 -- 172.21.15.155:0/3351573062 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2641678801 0 0) 0x7f6e3c003520 con 0x7f6e541430c0 2024-03-25T18:26:13.912 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.911+0000 7f6e52ffd640 1 -- 172.21.15.155:0/3351573062 >> v1:172.21.15.155:6790/0 conn(0x7f6e5413f710 legacy=0x7f6e541429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:13.912 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.911+0000 7f6e52ffd640 1 -- 172.21.15.155:0/3351573062 >> v1:172.21.15.155:6789/0 conn(0x7f6e540a98f0 legacy=0x7f6e540a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:13.913 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.911+0000 7f6e52ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:26:13.913 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.911+0000 7f6e52ffd640 1 monclient: found mon.noname-c 2024-03-25T18:26:13.913 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.911+0000 7f6e52ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:13.913 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.911+0000 7f6e52ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.167:6789/0 2024-03-25T18:26:13.913 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.911+0000 7f6e52ffd640 1 -- 172.21.15.155:0/3351573062 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e540b44c0 con 0x7f6e541430c0 2024-03-25T18:26:13.913 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.911+0000 7f6e52ffd640 10 monclient: _finish_auth 0 2024-03-25T18:26:13.913 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.911+0000 7f6e52ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:26:13.913 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.911+0000 7f6e52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:13.913 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.911+0000 7f6e52ffd640 1 -- 172.21.15.155:0/3351573062 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f6e4c001a10 con 0x7f6e541430c0 2024-03-25T18:26:13.913 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.911+0000 7f6e52ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:13.913 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.911+0000 7f6e52ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:13.913 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.911+0000 7f6e52ffd640 10 monclient: dump: 2024-03-25T18:26:13.913 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:26:13.913 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:26:13.913 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:13.914 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:13.914 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:13.914 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:26:13.914 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:26:13.914 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:26:13.914 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:26:13.914 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:26:13.914 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.911+0000 7f6e5a864640 5 monclient: authenticate success, global_id 4288 2024-03-25T18:26:13.914 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.911+0000 7f6e5a864640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:13.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.923+0000 7f6e52ffd640 1 -- 172.21.15.155:0/3351573062 <== mon.1 v1:172.21.15.167:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e4c001b70 con 0x7f6e541430c0 2024-03-25T18:26:13.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.923+0000 7f6e52ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:13.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.923+0000 7f6e52ffd640 1 -- 172.21.15.155:0/3351573062 <== mon.1 v1:172.21.15.167:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f6e4c001f80 con 0x7f6e541430c0 2024-03-25T18:26:13.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.923+0000 7f6e52ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:13.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.923+0000 7f6e52ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:13.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.923+0000 7f6e52ffd640 10 monclient: dump: 2024-03-25T18:26:13.924 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:26:13.924 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:26:13.924 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:13.924 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:13.924 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:13.924 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:26:13.924 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:26:13.925 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:26:13.925 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:26:13.925 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:26:13.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.923+0000 7f6e5a864640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:13.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.923+0000 7f6e5a864640 4 set_mon_vals no callback set 2024-03-25T18:26:13.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.923+0000 7f6e5a864640 10 monclient: shutdown 2024-03-25T18:26:13.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.923+0000 7f6e5a864640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:13.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.924+0000 7f6e5a864640 1 -- 172.21.15.155:0/3351573062 >> v1:172.21.15.167:6789/0 conn(0x7f6e541430c0 legacy=0x7f6e5414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:13.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.924+0000 7f6e5a864640 1 -- 172.21.15.155:0/3351573062 shutdown_connections 2024-03-25T18:26:13.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.924+0000 7f6e5a864640 1 -- 172.21.15.155:0/3351573062 >> 172.21.15.155:0/3351573062 conn(0x7f6e540b7ea0 msgr2=0x7f6e540b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:13.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.924+0000 7f6e5a864640 1 -- 172.21.15.155:0/3351573062 shutdown_connections 2024-03-25T18:26:13.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.924+0000 7f6e5a864640 1 -- 172.21.15.155:0/3351573062 wait complete. 2024-03-25T18:26:13.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.924+0000 7f6e5a864640 10 monclient: build_initial_monmap 2024-03-25T18:26:13.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.925+0000 7f6e5a864640 10 monclient: monmap: 2024-03-25T18:26:13.927 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:26:13.927 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:13.927 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:26:13.927 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:26:13.927 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:13.927 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:26:13.927 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:26:13.927 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:26:13.927 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:26:13.927 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:26:13.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.925+0000 7f6e5a864640 1 librados: starting msgr at 2024-03-25T18:26:13.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.925+0000 7f6e5a864640 1 librados: starting objecter 2024-03-25T18:26:13.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.925+0000 7f6e5a864640 1 Processor -- start 2024-03-25T18:26:13.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.925+0000 7f6e5a864640 1 -- start start 2024-03-25T18:26:13.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.925+0000 7f6e5a864640 1 librados: setting wanted keys 2024-03-25T18:26:13.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.925+0000 7f6e5a864640 1 librados: calling monclient init 2024-03-25T18:26:13.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.925+0000 7f6e5a864640 10 monclient: init 2024-03-25T18:26:13.929 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.926+0000 7f6e5a864640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:13.929 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.926+0000 7f6e5a864640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:26:13.929 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.926+0000 7f6e5a864640 10 monclient(hunting): picked mon.noname-c con 0x7f6e540a98f0 addr v1:172.21.15.155:6790/0 2024-03-25T18:26:13.929 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.926+0000 7f6e5a864640 10 monclient(hunting): picked mon.noname-b con 0x7f6e5413f710 addr v1:172.21.15.167:6789/0 2024-03-25T18:26:13.929 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.926+0000 7f6e5a864640 10 monclient(hunting): picked mon.noname-a con 0x7f6e541430c0 addr v1:172.21.15.155:6789/0 2024-03-25T18:26:13.929 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.926+0000 7f6e5a864640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e54136ad0 con 0x7f6e541430c0 2024-03-25T18:26:13.929 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.926+0000 7f6e5a864640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e540b7520 con 0x7f6e5413f710 2024-03-25T18:26:13.929 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.926+0000 7f6e5a864640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e54135810 con 0x7f6e540a98f0 2024-03-25T18:26:13.929 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.926+0000 7f6e5a864640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:13.929 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.926+0000 7f6e5a864640 10 monclient(hunting): authenticate will time out at 7620.236247s 2024-03-25T18:26:13.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.929+0000 7f6e537fe640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f6e5413f710 0x7f6e540b5a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:54518/0 (socket says 172.21.15.155:54518) 2024-03-25T18:26:13.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.929+0000 7f6e537fe640 1 -- 172.21.15.155:0/2899710212 learned_addr learned my addr 172.21.15.155:0/2899710212 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:26:13.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.930+0000 7f6e50ff9640 10 client.?.objecter ms_handle_connect 0x7f6e5413f710 2024-03-25T18:26:13.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.930+0000 7f6e50ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:13.940 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.939+0000 7f6e50ff9640 1 -- 172.21.15.155:0/2899710212 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1295559723 0 0) 0x7f6e540b7520 con 0x7f6e5413f710 2024-03-25T18:26:13.940 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.939+0000 7f6e50ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:26:13.940 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.939+0000 7f6e50ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:13.940 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.939+0000 7f6e50ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:13.940 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.939+0000 7f6e50ff9640 10 monclient(hunting): my global_id is 4291 2024-03-25T18:26:13.940 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.939+0000 7f6e50ff9640 1 -- 172.21.15.155:0/2899710212 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e40004eb0 con 0x7f6e5413f710 2024-03-25T18:26:13.949 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.948+0000 7f6e50ff9640 10 client.?.objecter ms_handle_connect 0x7f6e540a98f0 2024-03-25T18:26:13.949 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.948+0000 7f6e50ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:13.951 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.950+0000 7f6e50ff9640 1 -- 172.21.15.155:0/2899710212 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1148462741 0 0) 0x7f6e40004eb0 con 0x7f6e5413f710 2024-03-25T18:26:13.951 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.950+0000 7f6e50ff9640 1 -- 172.21.15.155:0/2899710212 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e540b7520 con 0x7f6e5413f710 2024-03-25T18:26:13.951 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.950+0000 7f6e50ff9640 1 -- 172.21.15.155:0/2899710212 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f6e480018b0 con 0x7f6e5413f710 2024-03-25T18:26:13.951 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.950+0000 7f6e50ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:13.951 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.950+0000 7f6e50ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:13.951 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.950+0000 7f6e50ff9640 10 monclient(hunting): dump: 2024-03-25T18:26:13.951 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:26:13.951 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:26:13.952 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:13.952 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:13.952 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:13.952 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:26:13.952 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:26:13.952 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:26:13.952 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:26:13.952 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:26:13.952 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.950+0000 7f6e50ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:13.958 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.957+0000 7f6e50ff9640 1 -- 172.21.15.155:0/2899710212 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3786857171 0 0) 0x7f6e54135810 con 0x7f6e540a98f0 2024-03-25T18:26:13.958 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.957+0000 7f6e50ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:26:13.958 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.957+0000 7f6e50ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:13.958 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.957+0000 7f6e50ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:13.958 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.957+0000 7f6e50ff9640 10 monclient(hunting): my global_id is 4292 2024-03-25T18:26:13.959 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.957+0000 7f6e50ff9640 1 -- 172.21.15.155:0/2899710212 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e40004eb0 con 0x7f6e540a98f0 2024-03-25T18:26:13.962 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.961+0000 7f6e50ff9640 1 -- 172.21.15.155:0/2899710212 <== mon.1 v1:172.21.15.167:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3002522553 0 0) 0x7f6e540b7520 con 0x7f6e5413f710 2024-03-25T18:26:13.962 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.961+0000 7f6e50ff9640 1 -- 172.21.15.155:0/2899710212 >> v1:172.21.15.155:6790/0 conn(0x7f6e540a98f0 legacy=0x7f6e54135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:13.962 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.961+0000 7f6e50ff9640 1 -- 172.21.15.155:0/2899710212 >> v1:172.21.15.155:6789/0 conn(0x7f6e541430c0 legacy=0x7f6e54130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:13.962 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.961+0000 7f6e50ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:26:13.962 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.961+0000 7f6e50ff9640 1 monclient: found mon.b 2024-03-25T18:26:13.962 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.961+0000 7f6e50ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:13.962 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.961+0000 7f6e50ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:26:13.963 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.961+0000 7f6e50ff9640 1 -- 172.21.15.155:0/2899710212 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e540b6180 con 0x7f6e5413f710 2024-03-25T18:26:13.963 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.961+0000 7f6e50ff9640 10 monclient: _finish_auth 0 2024-03-25T18:26:13.963 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.961+0000 7f6e50ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:26:13.963 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.961+0000 7f6e50ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:13.963 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.961+0000 7f6e5a864640 5 monclient: authenticate success, global_id 4291 2024-03-25T18:26:13.963 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.961+0000 7f6e5a864640 10 monclient: _renew_subs 2024-03-25T18:26:13.963 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.961+0000 7f6e5a864640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:26:13.963 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.961+0000 7f6e5a864640 1 -- 172.21.15.155:0/2899710212 --> v1:172.21.15.167:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e540b7a40 con 0x7f6e5413f710 2024-03-25T18:26:13.963 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.961+0000 7f6e5a864640 10 client.4291.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:13.963 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.961+0000 7f6e5a864640 10 monclient: _renew_subs 2024-03-25T18:26:13.963 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.961+0000 7f6e5a864640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:26:13.963 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.961+0000 7f6e5a864640 1 -- 172.21.15.155:0/2899710212 --> v1:172.21.15.167:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e54131840 con 0x7f6e5413f710 2024-03-25T18:26:13.963 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.961+0000 7f6e5a864640 1 librados: init done 2024-03-25T18:26:13.964 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.962+0000 7f6e5a864640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:13.964 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.962+0000 7f6e5a864640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:13.964 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.962+0000 7f6e5a864640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:26:13.964 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:13.962+0000 7f6e5a864640 1 -- 172.21.15.155:0/2899710212 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e54136f60 con 0x7f6e5413f710 2024-03-25T18:26:14.246 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:14.245+0000 7f6e50ff9640 1 -- 172.21.15.155:0/2899710212 <== mon.1 v1:172.21.15.167:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e480047e0 con 0x7f6e5413f710 2024-03-25T18:26:14.247 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:14.245+0000 7f6e50ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:14.247 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:14.245+0000 7f6e50ff9640 1 -- 172.21.15.155:0/2899710212 <== mon.1 v1:172.21.15.167:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f6e48004a50 con 0x7f6e5413f710 2024-03-25T18:26:14.247 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:14.245+0000 7f6e50ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:14.247 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:14.245+0000 7f6e50ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:14.247 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:14.245+0000 7f6e50ff9640 10 monclient: dump: 2024-03-25T18:26:14.247 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:26:14.247 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:26:14.247 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:14.247 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:14.247 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:14.247 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:26:14.247 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:26:14.247 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:26:14.247 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:26:14.248 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:26:14.248 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:14.245+0000 7f6e527fc640 4 set_mon_vals no callback set 2024-03-25T18:26:14.248 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:14.246+0000 7f6e50ff9640 1 -- 172.21.15.155:0/2899710212 <== mon.1 v1:172.21.15.167:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7f6e48004c70 con 0x7f6e5413f710 2024-03-25T18:26:14.248 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:14.247+0000 7f6e53fff640 10 monclient: get_auth_request con 0x7f6e4006dd30 auth_method 0 2024-03-25T18:26:14.248 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:14.247+0000 7f6e50ff9640 10 client.4291.objecter ms_handle_connect 0x7f6e4006dd30 2024-03-25T18:26:14.250 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:14.249+0000 7f6e50ff9640 1 -- 172.21.15.155:0/2899710212 <== mon.1 v1:172.21.15.167:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7f6e48005670 con 0x7f6e5413f710 2024-03-25T18:26:14.250 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:14.249+0000 7f6e50ff9640 10 client.4291.objecter ms_dispatch 0x7f6e54000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:14.250 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:14.249+0000 7f6e50ff9640 3 client.4291.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:14.250 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:14.249+0000 7f6e50ff9640 3 client.4291.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:14.250 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:14.249+0000 7f6e50ff9640 20 client.4291.objecter dump_active .. 0 homeless 2024-03-25T18:26:14.254 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:14.253+0000 7f6e50ff9640 1 -- 172.21.15.155:0/2899710212 <== mon.1 v1:172.21.15.167:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6e48050210 con 0x7f6e5413f710 2024-03-25T18:26:14.254 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:14.253+0000 7f6e50ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:14.254 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:14.253+0000 7f6e50ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:14.507 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:14.506+0000 7f6e5a864640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:14.507 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:14.506+0000 7f6e5a864640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:14.507 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:14.506+0000 7f6e5a864640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:26:14.507 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:14.506+0000 7f6e5a864640 1 -- 172.21.15.155:0/2899710212 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e540b6310 con 0x7f6e5413f710 2024-03-25T18:26:14.529 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:14.528+0000 7f6e50ff9640 1 -- 172.21.15.155:0/2899710212 <== mon.1 v1:172.21.15.167:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7f6e48052140 con 0x7f6e5413f710 2024-03-25T18:26:14.529 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:14.528+0000 7f6e50ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:14.529 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:14.528+0000 7f6e50ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:14.529 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:26:14.529 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:14.531 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:14.531+0000 7f6e5a864640 10 librados: watch_flush enter 2024-03-25T18:26:14.532 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:14.531+0000 7f6e5a864640 10 librados: watch_flush exit 2024-03-25T18:26:14.532 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:14.531+0000 7f6e5a864640 20 client.4291.objecter shutdown clearing up homeless session... 2024-03-25T18:26:14.532 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:14.531+0000 7f6e5a864640 10 client.4291.objecter successfully canceled tick 2024-03-25T18:26:14.532 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:14.531+0000 7f6e5a864640 1 -- 172.21.15.155:0/2899710212 >> v1:172.21.15.167:6800/2919070503 conn(0x7f6e4006dd30 legacy=0x7f6e400701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:14.532 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:14.531+0000 7f6e5a864640 10 monclient: shutdown 2024-03-25T18:26:14.532 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:14.531+0000 7f6e5a864640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:14.532 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:14.531+0000 7f6e5a864640 1 -- 172.21.15.155:0/2899710212 >> v1:172.21.15.167:6789/0 conn(0x7f6e5413f710 legacy=0x7f6e540b5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:14.533 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:14.532+0000 7f6e5a864640 1 -- 172.21.15.155:0/2899710212 shutdown_connections 2024-03-25T18:26:14.533 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:14.532+0000 7f6e5a864640 1 -- 172.21.15.155:0/2899710212 >> 172.21.15.155:0/2899710212 conn(0x7f6e540b7ea0 msgr2=0x7f6e54074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:14.533 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:14.532+0000 7f6e5a864640 1 -- 172.21.15.155:0/2899710212 shutdown_connections 2024-03-25T18:26:14.533 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:14.532+0000 7f6e5a864640 1 -- 172.21.15.155:0/2899710212 wait complete. 2024-03-25T18:26:14.533 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:14.532+0000 7f6e5a864640 1 librados: shutdown 2024-03-25T18:26:14.545 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:20.561 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:01:54.165 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:20.572 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:01:54.177 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:20.593 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:01:54.198 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:20.641 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:01:54.219 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:20.644 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:01:54.249 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:20.649 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:01:54.228 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:20.652 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:01:54.257 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:20.679 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:01:54.284 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:20.687 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:01:54.265 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:21.352 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:21.471 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.469+0000 7f5c17759640 10 monclient: get_monmap_and_config 2024-03-25T18:26:21.471 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.469+0000 7f5c17759640 10 monclient: build_initial_monmap 2024-03-25T18:26:21.472 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.469+0000 7f5c17759640 10 monclient: monmap: 2024-03-25T18:26:21.472 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:26:21.472 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:26:21.472 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:26:21.471741+0000 2024-03-25T18:26:21.472 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:26:21.471741+0000 2024-03-25T18:26:21.472 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:21.472 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:26:21.472 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:26:21.472 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:26:21.472 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:26:21.472 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:26:21.473 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.470+0000 7f5c17759640 1 Processor -- start 2024-03-25T18:26:21.473 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.470+0000 7f5c17759640 1 -- start start 2024-03-25T18:26:21.473 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.470+0000 7f5c17759640 10 monclient: init 2024-03-25T18:26:21.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.471+0000 7f5c17759640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:21.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.471+0000 7f5c17759640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:26:21.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.471+0000 7f5c17759640 10 monclient(hunting): picked mon.noname-b con 0x7f5c100a98f0 addr v1:172.21.15.155:6790/0 2024-03-25T18:26:21.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.471+0000 7f5c17759640 10 monclient(hunting): picked mon.noname-c con 0x7f5c1013f710 addr v1:172.21.15.167:6789/0 2024-03-25T18:26:21.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.471+0000 7f5c17759640 10 monclient(hunting): picked mon.noname-a con 0x7f5c101430c0 addr v1:172.21.15.155:6789/0 2024-03-25T18:26:21.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.471+0000 7f5c17759640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c10142520 con 0x7f5c101430c0 2024-03-25T18:26:21.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.471+0000 7f5c17759640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c100b28a0 con 0x7f5c1013f710 2024-03-25T18:26:21.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.471+0000 7f5c17759640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c100b2a30 con 0x7f5c100a98f0 2024-03-25T18:26:21.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.471+0000 7f5c17759640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:21.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.471+0000 7f5c17759640 10 monclient(hunting): authenticate will time out at 7627.782058s 2024-03-25T18:26:21.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.481+0000 7f5c154f5640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f5c100a98f0 0x7f5c100a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:38230/0 (socket says 172.21.15.155:38230) 2024-03-25T18:26:21.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.481+0000 7f5c154f5640 1 -- 172.21.15.155:0/2220007410 learned_addr learned my addr 172.21.15.155:0/2220007410 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:26:21.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.488+0000 7f5c07fff640 1 -- 172.21.15.155:0/2220007410 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2373333970 0 0) 0x7f5c100b2a30 con 0x7f5c100a98f0 2024-03-25T18:26:21.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.488+0000 7f5c07fff640 10 monclient(hunting): _negotiate 2024-03-25T18:26:21.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.488+0000 7f5c07fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:21.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.488+0000 7f5c07fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:21.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.488+0000 7f5c07fff640 10 monclient(hunting): my global_id is 4295 2024-03-25T18:26:21.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.488+0000 7f5c07fff640 1 -- 172.21.15.155:0/2220007410 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bfc003520 con 0x7f5c100a98f0 2024-03-25T18:26:21.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.498+0000 7f5c07fff640 1 -- 172.21.15.155:0/2220007410 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 629143497 0 0) 0x7f5bfc003520 con 0x7f5c100a98f0 2024-03-25T18:26:21.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.498+0000 7f5c07fff640 1 -- 172.21.15.155:0/2220007410 >> v1:172.21.15.167:6789/0 conn(0x7f5c1013f710 legacy=0x7f5c101429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:26:21.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.498+0000 7f5c07fff640 1 -- 172.21.15.155:0/2220007410 >> v1:172.21.15.155:6789/0 conn(0x7f5c101430c0 legacy=0x7f5c1014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:21.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.498+0000 7f5c07fff640 10 monclient: _finish_hunting 0 2024-03-25T18:26:21.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.498+0000 7f5c07fff640 1 monclient: found mon.noname-b 2024-03-25T18:26:21.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.498+0000 7f5c07fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:21.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.498+0000 7f5c07fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-25T18:26:21.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.498+0000 7f5c07fff640 1 -- 172.21.15.155:0/2220007410 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c100b44c0 con 0x7f5c100a98f0 2024-03-25T18:26:21.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.498+0000 7f5c07fff640 10 monclient: _finish_auth 0 2024-03-25T18:26:21.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.498+0000 7f5c07fff640 10 monclient: _check_auth_tickets 2024-03-25T18:26:21.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.498+0000 7f5c07fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:21.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.498+0000 7f5c17759640 5 monclient: authenticate success, global_id 4295 2024-03-25T18:26:21.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.498+0000 7f5c17759640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:21.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.498+0000 7f5c07fff640 1 -- 172.21.15.155:0/2220007410 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f5c0c001690 con 0x7f5c100a98f0 2024-03-25T18:26:21.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.498+0000 7f5c07fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:21.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.498+0000 7f5c07fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:21.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.498+0000 7f5c07fff640 10 monclient: dump: 2024-03-25T18:26:21.501 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:26:21.501 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:26:21.501 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:21.501 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:21.501 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:21.501 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:26:21.502 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:26:21.502 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:26:21.502 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:26:21.502 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:26:21.502 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.498+0000 7f5c17759640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:21.517 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.516+0000 7f5c07fff640 1 -- 172.21.15.155:0/2220007410 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5c0c001960 con 0x7f5c100a98f0 2024-03-25T18:26:21.518 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.516+0000 7f5c07fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:21.518 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.516+0000 7f5c07fff640 1 -- 172.21.15.155:0/2220007410 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f5c0c001cf0 con 0x7f5c100a98f0 2024-03-25T18:26:21.518 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.516+0000 7f5c07fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:21.518 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.516+0000 7f5c07fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:21.518 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.516+0000 7f5c07fff640 10 monclient: dump: 2024-03-25T18:26:21.518 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:26:21.518 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:26:21.518 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:21.518 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:21.518 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:21.518 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:26:21.519 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:26:21.519 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:26:21.519 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:26:21.519 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:26:21.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.516+0000 7f5c17759640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:21.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.516+0000 7f5c17759640 4 set_mon_vals no callback set 2024-03-25T18:26:21.521 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.516+0000 7f5c17759640 10 monclient: shutdown 2024-03-25T18:26:21.521 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.516+0000 7f5c17759640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:21.521 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.516+0000 7f5c17759640 1 -- 172.21.15.155:0/2220007410 >> v1:172.21.15.155:6790/0 conn(0x7f5c100a98f0 legacy=0x7f5c100a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:21.521 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.516+0000 7f5c17759640 1 -- 172.21.15.155:0/2220007410 shutdown_connections 2024-03-25T18:26:21.521 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.516+0000 7f5c17759640 1 -- 172.21.15.155:0/2220007410 >> 172.21.15.155:0/2220007410 conn(0x7f5c100b7ea0 msgr2=0x7f5c100b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:21.521 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.517+0000 7f5c17759640 1 -- 172.21.15.155:0/2220007410 shutdown_connections 2024-03-25T18:26:21.522 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.517+0000 7f5c17759640 1 -- 172.21.15.155:0/2220007410 wait complete. 2024-03-25T18:26:21.522 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.517+0000 7f5c17759640 10 monclient: build_initial_monmap 2024-03-25T18:26:21.522 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.517+0000 7f5c17759640 10 monclient: monmap: 2024-03-25T18:26:21.522 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:26:21.522 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:21.522 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:26:21.522 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:26:21.522 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:21.522 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:26:21.523 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:26:21.523 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:26:21.523 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:26:21.523 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:26:21.523 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.517+0000 7f5c17759640 1 librados: starting msgr at 2024-03-25T18:26:21.523 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.517+0000 7f5c17759640 1 librados: starting objecter 2024-03-25T18:26:21.523 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.518+0000 7f5c17759640 1 Processor -- start 2024-03-25T18:26:21.523 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.518+0000 7f5c17759640 1 -- start start 2024-03-25T18:26:21.523 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.518+0000 7f5c17759640 1 librados: setting wanted keys 2024-03-25T18:26:21.523 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.518+0000 7f5c17759640 1 librados: calling monclient init 2024-03-25T18:26:21.523 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.518+0000 7f5c17759640 10 monclient: init 2024-03-25T18:26:21.523 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.518+0000 7f5c17759640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:21.523 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.518+0000 7f5c17759640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:26:21.523 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.518+0000 7f5c17759640 10 monclient(hunting): picked mon.noname-b con 0x7f5c100a98f0 addr v1:172.21.15.167:6789/0 2024-03-25T18:26:21.523 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.518+0000 7f5c17759640 10 monclient(hunting): picked mon.noname-a con 0x7f5c1013f710 addr v1:172.21.15.155:6789/0 2024-03-25T18:26:21.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.518+0000 7f5c17759640 10 monclient(hunting): picked mon.noname-c con 0x7f5c101430c0 addr v1:172.21.15.155:6790/0 2024-03-25T18:26:21.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.519+0000 7f5c17759640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c100b1100 con 0x7f5c1013f710 2024-03-25T18:26:21.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.519+0000 7f5c17759640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c10137a60 con 0x7f5c100a98f0 2024-03-25T18:26:21.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.519+0000 7f5c17759640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c10137bf0 con 0x7f5c101430c0 2024-03-25T18:26:21.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.519+0000 7f5c17759640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:21.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.519+0000 7f5c17759640 10 monclient(hunting): authenticate will time out at 7627.829838s 2024-03-25T18:26:21.536 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.534+0000 7f5c15cf6640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f5c101430c0 0x7f5c101e1f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:38262/0 (socket says 172.21.15.155:38262) 2024-03-25T18:26:21.536 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.534+0000 7f5c15cf6640 1 -- 172.21.15.155:0/4287993747 learned_addr learned my addr 172.21.15.155:0/4287993747 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:26:21.537 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.534+0000 7f5c05ffb640 10 client.?.objecter ms_handle_connect 0x7f5c101430c0 2024-03-25T18:26:21.537 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.534+0000 7f5c05ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:21.541 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.539+0000 7f5c05ffb640 10 client.?.objecter ms_handle_connect 0x7f5c1013f710 2024-03-25T18:26:21.541 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.539+0000 7f5c05ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:21.542 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.540+0000 7f5c05ffb640 10 client.?.objecter ms_handle_connect 0x7f5c100a98f0 2024-03-25T18:26:21.542 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.540+0000 7f5c05ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:21.550 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.548+0000 7f5c05ffb640 1 -- 172.21.15.155:0/4287993747 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1111564398 0 0) 0x7f5c10137bf0 con 0x7f5c101430c0 2024-03-25T18:26:21.550 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.548+0000 7f5c05ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:26:21.550 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.548+0000 7f5c05ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:21.550 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.548+0000 7f5c05ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:21.551 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.548+0000 7f5c05ffb640 10 monclient(hunting): my global_id is 4298 2024-03-25T18:26:21.551 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.549+0000 7f5c05ffb640 1 -- 172.21.15.155:0/4287993747 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bf4004eb0 con 0x7f5c101430c0 2024-03-25T18:26:21.553 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.551+0000 7f5c05ffb640 1 -- 172.21.15.155:0/4287993747 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 238996536 0 0) 0x7f5c100b1100 con 0x7f5c1013f710 2024-03-25T18:26:21.553 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.551+0000 7f5c05ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:26:21.553 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.551+0000 7f5c05ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:21.553 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.551+0000 7f5c05ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:21.553 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.551+0000 7f5c05ffb640 10 monclient(hunting): my global_id is 4293 2024-03-25T18:26:21.553 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.551+0000 7f5c05ffb640 1 -- 172.21.15.155:0/4287993747 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c10137bf0 con 0x7f5c1013f710 2024-03-25T18:26:21.566 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.564+0000 7f5c05ffb640 1 -- 172.21.15.155:0/4287993747 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3683431389 0 0) 0x7f5c10137bf0 con 0x7f5c1013f710 2024-03-25T18:26:21.566 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.564+0000 7f5c05ffb640 1 -- 172.21.15.155:0/4287993747 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c100b1100 con 0x7f5c1013f710 2024-03-25T18:26:21.566 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.564+0000 7f5c05ffb640 1 -- 172.21.15.155:0/4287993747 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f5c00001510 con 0x7f5c1013f710 2024-03-25T18:26:21.566 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.564+0000 7f5c05ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:21.566 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.564+0000 7f5c05ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:26:21.566 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.564+0000 7f5c05ffb640 10 monclient(hunting): dump: 2024-03-25T18:26:21.566 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:26:21.566 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:26:21.566 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:21.567 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:21.567 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:21.567 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:26:21.567 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:26:21.567 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:26:21.567 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:26:21.567 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:26:21.567 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.564+0000 7f5c05ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:21.583 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.581+0000 7f5c05ffb640 1 -- 172.21.15.155:0/4287993747 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2395613438 0 0) 0x7f5c100b1100 con 0x7f5c1013f710 2024-03-25T18:26:21.583 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.581+0000 7f5c05ffb640 1 -- 172.21.15.155:0/4287993747 >> v1:172.21.15.155:6790/0 conn(0x7f5c101430c0 legacy=0x7f5c101e1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:21.584 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.581+0000 7f5c05ffb640 1 -- 172.21.15.155:0/4287993747 >> v1:172.21.15.167:6789/0 conn(0x7f5c100a98f0 legacy=0x7f5c1013b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:21.584 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.581+0000 7f5c05ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:26:21.584 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.581+0000 7f5c05ffb640 1 monclient: found mon.a 2024-03-25T18:26:21.584 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.581+0000 7f5c05ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:21.584 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.581+0000 7f5c05ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:26:21.584 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.581+0000 7f5c05ffb640 1 -- 172.21.15.155:0/4287993747 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c100b1590 con 0x7f5c1013f710 2024-03-25T18:26:21.584 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.581+0000 7f5c05ffb640 10 monclient: _finish_auth 0 2024-03-25T18:26:21.584 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.581+0000 7f5c05ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:26:21.584 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.581+0000 7f5c05ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:21.584 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.581+0000 7f5c17759640 5 monclient: authenticate success, global_id 4293 2024-03-25T18:26:21.584 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.581+0000 7f5c17759640 10 monclient: _renew_subs 2024-03-25T18:26:21.584 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.581+0000 7f5c17759640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:26:21.584 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.581+0000 7f5c17759640 1 -- 172.21.15.155:0/4287993747 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c100b1780 con 0x7f5c1013f710 2024-03-25T18:26:21.585 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.581+0000 7f5c17759640 10 client.4293.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:21.585 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.581+0000 7f5c17759640 10 monclient: _renew_subs 2024-03-25T18:26:21.585 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.581+0000 7f5c17759640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:26:21.585 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.581+0000 7f5c17759640 1 -- 172.21.15.155:0/4287993747 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c10136b50 con 0x7f5c1013f710 2024-03-25T18:26:21.585 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.581+0000 7f5c17759640 1 librados: init done 2024-03-25T18:26:21.585 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.582+0000 7f5c17759640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:21.586 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.582+0000 7f5c17759640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:21.586 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.582+0000 7f5c17759640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:26:21.586 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.582+0000 7f5c17759640 1 -- 172.21.15.155:0/4287993747 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c100b12b0 con 0x7f5c1013f710 2024-03-25T18:26:21.962 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.959+0000 7f5c05ffb640 1 -- 172.21.15.155:0/4287993747 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5c00001a00 con 0x7f5c1013f710 2024-03-25T18:26:21.962 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.959+0000 7f5c05ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:21.962 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.959+0000 7f5c05ffb640 1 -- 172.21.15.155:0/4287993747 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f5c00001bf0 con 0x7f5c1013f710 2024-03-25T18:26:21.962 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.959+0000 7f5c05ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:21.962 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.959+0000 7f5c05ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:21.962 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.959+0000 7f5c05ffb640 10 monclient: dump: 2024-03-25T18:26:21.962 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:26:21.962 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:26:21.963 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:21.963 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:21.963 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:21.963 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:26:21.963 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:26:21.963 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:26:21.963 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:26:21.963 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:26:21.963 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.959+0000 7f5c077fe640 4 set_mon_vals no callback set 2024-03-25T18:26:21.963 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.960+0000 7f5c05ffb640 1 -- 172.21.15.155:0/4287993747 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7f5c00019780 con 0x7f5c1013f710 2024-03-25T18:26:21.964 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.961+0000 7f5c154f5640 10 monclient: get_auth_request con 0x7f5bf406de20 auth_method 0 2024-03-25T18:26:21.964 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.961+0000 7f5c05ffb640 10 client.4293.objecter ms_handle_connect 0x7f5bf406de20 2024-03-25T18:26:21.969 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.967+0000 7f5c05ffb640 1 -- 172.21.15.155:0/4287993747 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7f5c000532e0 con 0x7f5c1013f710 2024-03-25T18:26:21.969 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.967+0000 7f5c05ffb640 10 client.4293.objecter ms_dispatch 0x7f5c10000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:21.969 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.967+0000 7f5c05ffb640 3 client.4293.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:21.970 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.967+0000 7f5c05ffb640 3 client.4293.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:21.970 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.967+0000 7f5c05ffb640 20 client.4293.objecter dump_active .. 0 homeless 2024-03-25T18:26:21.970 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.967+0000 7f5c05ffb640 1 -- 172.21.15.155:0/4287993747 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5c0004edf0 con 0x7f5c1013f710 2024-03-25T18:26:21.970 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.967+0000 7f5c05ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:21.970 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:21.967+0000 7f5c05ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:22.179 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:22.177+0000 7f5c17759640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:22.179 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:22.177+0000 7f5c17759640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:22.179 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:22.177+0000 7f5c17759640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:26:22.179 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:22.177+0000 7f5c17759640 1 -- 172.21.15.155:0/4287993747 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5c100627b0 con 0x7f5c1013f710 2024-03-25T18:26:22.226 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:22.223+0000 7f5c05ffb640 1 -- 172.21.15.155:0/4287993747 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7f5c00050d20 con 0x7f5c1013f710 2024-03-25T18:26:22.226 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:22.224+0000 7f5c05ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:22.226 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:22.224+0000 7f5c05ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:22.226 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:26:22.226 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:22.228 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:22.226+0000 7f5c17759640 10 librados: watch_flush enter 2024-03-25T18:26:22.229 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:22.226+0000 7f5c17759640 10 librados: watch_flush exit 2024-03-25T18:26:22.229 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:22.226+0000 7f5c17759640 20 client.4293.objecter shutdown clearing up homeless session... 2024-03-25T18:26:22.229 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:22.226+0000 7f5c17759640 10 client.4293.objecter successfully canceled tick 2024-03-25T18:26:22.229 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:22.226+0000 7f5c17759640 1 -- 172.21.15.155:0/4287993747 >> v1:172.21.15.167:6800/2919070503 conn(0x7f5bf406de20 legacy=0x7f5bf40702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:22.229 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:22.226+0000 7f5c17759640 10 monclient: shutdown 2024-03-25T18:26:22.229 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:22.226+0000 7f5c17759640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:22.229 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:22.226+0000 7f5c17759640 1 -- 172.21.15.155:0/4287993747 >> v1:172.21.15.155:6789/0 conn(0x7f5c1013f710 legacy=0x7f5c10137350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:22.229 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:22.226+0000 7f5c17759640 1 -- 172.21.15.155:0/4287993747 shutdown_connections 2024-03-25T18:26:22.229 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:22.226+0000 7f5c17759640 1 -- 172.21.15.155:0/4287993747 >> 172.21.15.155:0/4287993747 conn(0x7f5c100b7ea0 msgr2=0x7f5c10074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:22.229 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:22.226+0000 7f5c17759640 1 -- 172.21.15.155:0/4287993747 shutdown_connections 2024-03-25T18:26:22.229 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:22.226+0000 7f5c17759640 1 -- 172.21.15.155:0/4287993747 wait complete. 2024-03-25T18:26:22.229 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:22.226+0000 7f5c17759640 1 librados: shutdown 2024-03-25T18:26:22.240 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:29.048 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:29.165 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.164+0000 7f8ccaad4640 10 monclient: get_monmap_and_config 2024-03-25T18:26:29.165 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.164+0000 7f8ccaad4640 10 monclient: build_initial_monmap 2024-03-25T18:26:29.166 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.164+0000 7f8ccaad4640 10 monclient: monmap: 2024-03-25T18:26:29.166 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:26:29.166 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:26:29.166 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:26:29.165708+0000 2024-03-25T18:26:29.166 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:26:29.165708+0000 2024-03-25T18:26:29.166 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:29.166 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:26:29.166 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:26:29.166 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:26:29.166 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:26:29.166 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:26:29.168 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.165+0000 7f8ccaad4640 1 Processor -- start 2024-03-25T18:26:29.168 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.165+0000 7f8ccaad4640 1 -- start start 2024-03-25T18:26:29.168 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.165+0000 7f8ccaad4640 10 monclient: init 2024-03-25T18:26:29.168 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.166+0000 7f8ccaad4640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:29.168 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.166+0000 7f8ccaad4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:26:29.168 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.166+0000 7f8ccaad4640 10 monclient(hunting): picked mon.noname-a con 0x7f8cc40a98f0 addr v1:172.21.15.155:6789/0 2024-03-25T18:26:29.168 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.166+0000 7f8ccaad4640 10 monclient(hunting): picked mon.noname-b con 0x7f8cc413f710 addr v1:172.21.15.155:6790/0 2024-03-25T18:26:29.168 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.166+0000 7f8ccaad4640 10 monclient(hunting): picked mon.noname-c con 0x7f8cc41430c0 addr v1:172.21.15.167:6789/0 2024-03-25T18:26:29.168 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.166+0000 7f8ccaad4640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cc4142520 con 0x7f8cc40a98f0 2024-03-25T18:26:29.168 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.166+0000 7f8ccaad4640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cc40b28a0 con 0x7f8cc41430c0 2024-03-25T18:26:29.168 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.166+0000 7f8ccaad4640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cc40b2a30 con 0x7f8cc413f710 2024-03-25T18:26:29.168 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.166+0000 7f8ccaad4640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:29.168 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.166+0000 7f8ccaad4640 10 monclient(hunting): authenticate will time out at 7635.475996s 2024-03-25T18:26:29.178 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.177+0000 7f8cc9071640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f8cc41430c0 0x7f8cc414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:59322/0 (socket says 172.21.15.155:59322) 2024-03-25T18:26:29.178 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.177+0000 7f8cc9071640 1 -- 172.21.15.155:0/3461050498 learned_addr learned my addr 172.21.15.155:0/3461050498 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:26:29.191 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.190+0000 7f8cc37fe640 1 -- 172.21.15.155:0/3461050498 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1925264197 0 0) 0x7f8cc40b28a0 con 0x7f8cc41430c0 2024-03-25T18:26:29.191 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.190+0000 7f8cc37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:26:29.191 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.190+0000 7f8cc37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:29.191 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.190+0000 7f8cc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:29.191 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.190+0000 7f8cc37fe640 10 monclient(hunting): my global_id is 4294 2024-03-25T18:26:29.191 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.190+0000 7f8cc37fe640 1 -- 172.21.15.155:0/3461050498 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cb4003520 con 0x7f8cc41430c0 2024-03-25T18:26:29.198 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.197+0000 7f8cc37fe640 1 -- 172.21.15.155:0/3461050498 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2785873403 0 0) 0x7f8cc40b2a30 con 0x7f8cc413f710 2024-03-25T18:26:29.198 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.197+0000 7f8cc37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:26:29.198 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.197+0000 7f8cc37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:29.198 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.197+0000 7f8cc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:29.198 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.197+0000 7f8cc37fe640 10 monclient(hunting): my global_id is 4301 2024-03-25T18:26:29.199 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.197+0000 7f8cc37fe640 1 -- 172.21.15.155:0/3461050498 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cc40b28a0 con 0x7f8cc413f710 2024-03-25T18:26:29.199 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.197+0000 7f8cc37fe640 1 -- 172.21.15.155:0/3461050498 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2846836788 0 0) 0x7f8cc4142520 con 0x7f8cc40a98f0 2024-03-25T18:26:29.199 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.197+0000 7f8cc37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:26:29.199 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.197+0000 7f8cc37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:29.199 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.197+0000 7f8cc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:29.199 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.197+0000 7f8cc37fe640 10 monclient(hunting): my global_id is 4296 2024-03-25T18:26:29.199 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.197+0000 7f8cc37fe640 1 -- 172.21.15.155:0/3461050498 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cc40b2a30 con 0x7f8cc40a98f0 2024-03-25T18:26:29.200 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.199+0000 7f8cc37fe640 1 -- 172.21.15.155:0/3461050498 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3381954198 0 0) 0x7f8cb4003520 con 0x7f8cc41430c0 2024-03-25T18:26:29.201 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.199+0000 7f8cc37fe640 1 -- 172.21.15.155:0/3461050498 >> v1:172.21.15.155:6790/0 conn(0x7f8cc413f710 legacy=0x7f8cc41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:29.201 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.199+0000 7f8cc37fe640 1 -- 172.21.15.155:0/3461050498 >> v1:172.21.15.155:6789/0 conn(0x7f8cc40a98f0 legacy=0x7f8cc40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:29.201 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.199+0000 7f8cc37fe640 10 monclient: _finish_hunting 0 2024-03-25T18:26:29.201 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.199+0000 7f8cc37fe640 1 monclient: found mon.noname-c 2024-03-25T18:26:29.201 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.199+0000 7f8cc37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:29.201 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.199+0000 7f8cc37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.167:6789/0 2024-03-25T18:26:29.201 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.199+0000 7f8cc37fe640 1 -- 172.21.15.155:0/3461050498 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc40b44c0 con 0x7f8cc41430c0 2024-03-25T18:26:29.201 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.199+0000 7f8cc37fe640 10 monclient: _finish_auth 0 2024-03-25T18:26:29.201 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.199+0000 7f8cc37fe640 10 monclient: _check_auth_tickets 2024-03-25T18:26:29.201 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.199+0000 7f8cc37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:29.201 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.199+0000 7f8cc37fe640 1 -- 172.21.15.155:0/3461050498 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f8cbc001a10 con 0x7f8cc41430c0 2024-03-25T18:26:29.201 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.199+0000 7f8ccaad4640 5 monclient: authenticate success, global_id 4294 2024-03-25T18:26:29.202 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.199+0000 7f8ccaad4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:29.202 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.199+0000 7f8cc37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:29.202 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.199+0000 7f8cc37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:29.202 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.199+0000 7f8cc37fe640 10 monclient: dump: 2024-03-25T18:26:29.202 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:26:29.202 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:26:29.202 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:29.202 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:29.202 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:29.202 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:26:29.202 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:26:29.202 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:26:29.202 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:26:29.202 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:26:29.203 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.199+0000 7f8ccaad4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:29.211 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.210+0000 7f8cc37fe640 1 -- 172.21.15.155:0/3461050498 <== mon.1 v1:172.21.15.167:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8cbc001b70 con 0x7f8cc41430c0 2024-03-25T18:26:29.211 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.210+0000 7f8cc37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:29.211 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.210+0000 7f8cc37fe640 1 -- 172.21.15.155:0/3461050498 <== mon.1 v1:172.21.15.167:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f8cbc001f80 con 0x7f8cc41430c0 2024-03-25T18:26:29.211 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.210+0000 7f8cc37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:29.211 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.210+0000 7f8cc37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:29.211 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.210+0000 7f8cc37fe640 10 monclient: dump: 2024-03-25T18:26:29.211 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:26:29.211 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:26:29.211 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:29.211 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:29.212 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:29.212 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:26:29.212 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:26:29.212 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:26:29.212 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:26:29.212 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:26:29.212 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.210+0000 7f8ccaad4640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:29.212 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.210+0000 7f8ccaad4640 4 set_mon_vals no callback set 2024-03-25T18:26:29.213 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.211+0000 7f8ccaad4640 10 monclient: shutdown 2024-03-25T18:26:29.213 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.211+0000 7f8ccaad4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:29.213 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.211+0000 7f8ccaad4640 1 -- 172.21.15.155:0/3461050498 >> v1:172.21.15.167:6789/0 conn(0x7f8cc41430c0 legacy=0x7f8cc414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:29.214 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.211+0000 7f8ccaad4640 1 -- 172.21.15.155:0/3461050498 shutdown_connections 2024-03-25T18:26:29.214 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.211+0000 7f8ccaad4640 1 -- 172.21.15.155:0/3461050498 >> 172.21.15.155:0/3461050498 conn(0x7f8cc40b7ea0 msgr2=0x7f8cc40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:29.214 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.211+0000 7f8ccaad4640 1 -- 172.21.15.155:0/3461050498 shutdown_connections 2024-03-25T18:26:29.214 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.211+0000 7f8ccaad4640 1 -- 172.21.15.155:0/3461050498 wait complete. 2024-03-25T18:26:29.214 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.212+0000 7f8ccaad4640 10 monclient: build_initial_monmap 2024-03-25T18:26:29.214 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.212+0000 7f8ccaad4640 10 monclient: monmap: 2024-03-25T18:26:29.214 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:26:29.214 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:29.214 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:26:29.214 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:26:29.214 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:29.214 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:26:29.215 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:26:29.215 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:26:29.215 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:26:29.215 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:26:29.215 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.212+0000 7f8ccaad4640 1 librados: starting msgr at 2024-03-25T18:26:29.215 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.212+0000 7f8ccaad4640 1 librados: starting objecter 2024-03-25T18:26:29.215 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.212+0000 7f8ccaad4640 1 Processor -- start 2024-03-25T18:26:29.215 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.212+0000 7f8ccaad4640 1 -- start start 2024-03-25T18:26:29.215 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.212+0000 7f8ccaad4640 1 librados: setting wanted keys 2024-03-25T18:26:29.215 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.212+0000 7f8ccaad4640 1 librados: calling monclient init 2024-03-25T18:26:29.215 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.212+0000 7f8ccaad4640 10 monclient: init 2024-03-25T18:26:29.216 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.213+0000 7f8ccaad4640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:29.216 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.213+0000 7f8ccaad4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:26:29.216 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.213+0000 7f8ccaad4640 10 monclient(hunting): picked mon.noname-b con 0x7f8cc40a98f0 addr v1:172.21.15.167:6789/0 2024-03-25T18:26:29.217 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.213+0000 7f8ccaad4640 10 monclient(hunting): picked mon.noname-c con 0x7f8cc413f710 addr v1:172.21.15.155:6790/0 2024-03-25T18:26:29.217 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.214+0000 7f8ccaad4640 10 monclient(hunting): picked mon.noname-a con 0x7f8cc41430c0 addr v1:172.21.15.155:6789/0 2024-03-25T18:26:29.217 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.214+0000 7f8ccaad4640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cc41daca0 con 0x7f8cc41430c0 2024-03-25T18:26:29.217 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.214+0000 7f8ccaad4640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cc4138ab0 con 0x7f8cc40a98f0 2024-03-25T18:26:29.217 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.214+0000 7f8ccaad4640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cc40aba90 con 0x7f8cc413f710 2024-03-25T18:26:29.217 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.214+0000 7f8ccaad4640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:29.217 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.214+0000 7f8ccaad4640 10 monclient(hunting): authenticate will time out at 7635.523888s 2024-03-25T18:26:29.223 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.222+0000 7f8cc8870640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f8cc40a98f0 0x7f8cc41383a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:59386/0 (socket says 172.21.15.155:59386) 2024-03-25T18:26:29.223 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.222+0000 7f8cc8870640 1 -- 172.21.15.155:0/827496220 learned_addr learned my addr 172.21.15.155:0/827496220 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:26:29.229 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.228+0000 7f8cc17fa640 10 client.?.objecter ms_handle_connect 0x7f8cc41430c0 2024-03-25T18:26:29.229 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.228+0000 7f8cc17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:29.229 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.228+0000 7f8cc17fa640 10 client.?.objecter ms_handle_connect 0x7f8cc40a98f0 2024-03-25T18:26:29.229 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.228+0000 7f8cc17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:29.230 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.229+0000 7f8cc17fa640 10 client.?.objecter ms_handle_connect 0x7f8cc413f710 2024-03-25T18:26:29.230 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.229+0000 7f8cc17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:29.238 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.237+0000 7f8cc17fa640 1 -- 172.21.15.155:0/827496220 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3339454007 0 0) 0x7f8cc41daca0 con 0x7f8cc41430c0 2024-03-25T18:26:29.238 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.237+0000 7f8cc17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:26:29.238 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.237+0000 7f8cc17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:29.241 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.237+0000 7f8cc17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:29.242 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.237+0000 7f8cc17fa640 10 monclient(hunting): my global_id is 4299 2024-03-25T18:26:29.242 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.237+0000 7f8cc17fa640 1 -- 172.21.15.155:0/827496220 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ca8004eb0 con 0x7f8cc41430c0 2024-03-25T18:26:29.242 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.241+0000 7f8cc17fa640 1 -- 172.21.15.155:0/827496220 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4200025292 0 0) 0x7f8cc4138ab0 con 0x7f8cc40a98f0 2024-03-25T18:26:29.242 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.241+0000 7f8cc17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:26:29.242 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.241+0000 7f8cc17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:29.242 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.241+0000 7f8cc17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:29.242 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.241+0000 7f8cc17fa640 10 monclient(hunting): my global_id is 4297 2024-03-25T18:26:29.243 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.241+0000 7f8cc17fa640 1 -- 172.21.15.155:0/827496220 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cc41daca0 con 0x7f8cc40a98f0 2024-03-25T18:26:29.249 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.249+0000 7f8cc17fa640 1 -- 172.21.15.155:0/827496220 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1080623196 0 0) 0x7f8cc40aba90 con 0x7f8cc413f710 2024-03-25T18:26:29.249 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.249+0000 7f8cc17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:26:29.249 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.249+0000 7f8cc17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:29.250 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.249+0000 7f8cc17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:29.250 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.249+0000 7f8cc17fa640 10 monclient(hunting): my global_id is 4304 2024-03-25T18:26:29.250 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.249+0000 7f8cc17fa640 1 -- 172.21.15.155:0/827496220 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cc4138ab0 con 0x7f8cc413f710 2024-03-25T18:26:29.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.266+0000 7f8cc17fa640 1 -- 172.21.15.155:0/827496220 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1560321623 0 0) 0x7f8ca8004eb0 con 0x7f8cc41430c0 2024-03-25T18:26:29.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.266+0000 7f8cc17fa640 1 -- 172.21.15.155:0/827496220 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8cc40aba90 con 0x7f8cc41430c0 2024-03-25T18:26:29.268 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.266+0000 7f8cc17fa640 1 -- 172.21.15.155:0/827496220 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f8cbc001e80 con 0x7f8cc41430c0 2024-03-25T18:26:29.268 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.266+0000 7f8cc17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:29.268 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.266+0000 7f8cc17fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:26:29.268 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.266+0000 7f8cc17fa640 10 monclient(hunting): dump: 2024-03-25T18:26:29.268 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:26:29.268 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:26:29.268 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:29.268 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:29.268 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:29.268 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:26:29.268 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:26:29.268 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:26:29.269 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:26:29.269 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:26:29.269 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.266+0000 7f8cc17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:29.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.276+0000 7f8cc17fa640 1 -- 172.21.15.155:0/827496220 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3531591661 0 0) 0x7f8cc40aba90 con 0x7f8cc41430c0 2024-03-25T18:26:29.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.276+0000 7f8cc17fa640 1 -- 172.21.15.155:0/827496220 >> v1:172.21.15.155:6790/0 conn(0x7f8cc413f710 legacy=0x7f8cc41db130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:29.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.276+0000 7f8cc17fa640 1 -- 172.21.15.155:0/827496220 >> v1:172.21.15.167:6789/0 conn(0x7f8cc40a98f0 legacy=0x7f8cc41383a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:29.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.276+0000 7f8cc17fa640 10 monclient: _finish_hunting 0 2024-03-25T18:26:29.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.276+0000 7f8cc17fa640 1 monclient: found mon.a 2024-03-25T18:26:29.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.276+0000 7f8cc17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:29.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.276+0000 7f8cc17fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:26:29.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.276+0000 7f8cc17fa640 1 -- 172.21.15.155:0/827496220 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc41d41e0 con 0x7f8cc41430c0 2024-03-25T18:26:29.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.276+0000 7f8cc17fa640 10 monclient: _finish_auth 0 2024-03-25T18:26:29.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.276+0000 7f8cc17fa640 10 monclient: _check_auth_tickets 2024-03-25T18:26:29.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.276+0000 7f8cc17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:29.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.276+0000 7f8ccaad4640 5 monclient: authenticate success, global_id 4299 2024-03-25T18:26:29.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.276+0000 7f8ccaad4640 10 monclient: _renew_subs 2024-03-25T18:26:29.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.276+0000 7f8ccaad4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:26:29.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.276+0000 7f8ccaad4640 1 -- 172.21.15.155:0/827496220 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cc41db840 con 0x7f8cc41430c0 2024-03-25T18:26:29.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.276+0000 7f8ccaad4640 10 client.4299.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:29.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.276+0000 7f8ccaad4640 10 monclient: _renew_subs 2024-03-25T18:26:29.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.276+0000 7f8ccaad4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:26:29.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.276+0000 7f8ccaad4640 1 -- 172.21.15.155:0/827496220 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cc4138f00 con 0x7f8cc41430c0 2024-03-25T18:26:29.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.276+0000 7f8ccaad4640 1 librados: init done 2024-03-25T18:26:29.280 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.278+0000 7f8ccaad4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:29.280 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.278+0000 7f8ccaad4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:29.280 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.278+0000 7f8ccaad4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:26:29.280 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.278+0000 7f8ccaad4640 1 -- 172.21.15.155:0/827496220 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cc41dae50 con 0x7f8cc41430c0 2024-03-25T18:26:29.586 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.585+0000 7f8cc17fa640 1 -- 172.21.15.155:0/827496220 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8cbc002020 con 0x7f8cc41430c0 2024-03-25T18:26:29.587 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.585+0000 7f8cc17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:29.587 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.585+0000 7f8cc17fa640 1 -- 172.21.15.155:0/827496220 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f8cbc004e10 con 0x7f8cc41430c0 2024-03-25T18:26:29.587 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.585+0000 7f8cc17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:29.587 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.585+0000 7f8cc17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:29.587 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.585+0000 7f8cc17fa640 10 monclient: dump: 2024-03-25T18:26:29.587 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:26:29.587 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:26:29.587 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:29.587 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:29.587 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:29.587 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:26:29.587 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:26:29.588 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:26:29.588 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:26:29.588 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:26:29.588 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.585+0000 7f8cc2ffd640 4 set_mon_vals no callback set 2024-03-25T18:26:29.588 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.586+0000 7f8cc17fa640 1 -- 172.21.15.155:0/827496220 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7f8cbc01ade0 con 0x7f8cc41430c0 2024-03-25T18:26:29.588 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.587+0000 7f8cc8870640 10 monclient: get_auth_request con 0x7f8ca806dda0 auth_method 0 2024-03-25T18:26:29.588 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.587+0000 7f8cc17fa640 10 client.4299.objecter ms_handle_connect 0x7f8ca806dda0 2024-03-25T18:26:29.595 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.594+0000 7f8cc17fa640 1 -- 172.21.15.155:0/827496220 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7f8cbc0549b0 con 0x7f8cc41430c0 2024-03-25T18:26:29.595 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.594+0000 7f8cc17fa640 10 client.4299.objecter ms_dispatch 0x7f8cc4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:29.595 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.594+0000 7f8cc17fa640 3 client.4299.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:29.595 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.594+0000 7f8cc17fa640 3 client.4299.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:29.595 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.594+0000 7f8cc17fa640 20 client.4299.objecter dump_active .. 0 homeless 2024-03-25T18:26:29.595 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.594+0000 7f8cc17fa640 1 -- 172.21.15.155:0/827496220 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8cbc0502f0 con 0x7f8cc41430c0 2024-03-25T18:26:29.595 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.594+0000 7f8cc17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:29.595 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.594+0000 7f8cc17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:29.826 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.825+0000 7f8ccaad4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:29.826 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.825+0000 7f8ccaad4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:29.826 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.825+0000 7f8ccaad4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:26:29.826 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.825+0000 7f8ccaad4640 1 -- 172.21.15.155:0/827496220 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8cc40627b0 con 0x7f8cc41430c0 2024-03-25T18:26:29.845 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.844+0000 7f8cc17fa640 1 -- 172.21.15.155:0/827496220 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7f8cbc0523f0 con 0x7f8cc41430c0 2024-03-25T18:26:29.845 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.844+0000 7f8cc17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:29.845 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.844+0000 7f8cc17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:29.846 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:26:29.846 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:29.848 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.847+0000 7f8ccaad4640 10 librados: watch_flush enter 2024-03-25T18:26:29.848 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.847+0000 7f8ccaad4640 10 librados: watch_flush exit 2024-03-25T18:26:29.849 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.847+0000 7f8ccaad4640 20 client.4299.objecter shutdown clearing up homeless session... 2024-03-25T18:26:29.849 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.847+0000 7f8ccaad4640 10 client.4299.objecter successfully canceled tick 2024-03-25T18:26:29.849 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.847+0000 7f8ccaad4640 1 -- 172.21.15.155:0/827496220 >> v1:172.21.15.167:6800/2919070503 conn(0x7f8ca806dda0 legacy=0x7f8ca8070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:29.849 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.847+0000 7f8ccaad4640 10 monclient: shutdown 2024-03-25T18:26:29.849 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.847+0000 7f8ccaad4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:29.849 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.847+0000 7f8ccaad4640 1 -- 172.21.15.155:0/827496220 >> v1:172.21.15.155:6789/0 conn(0x7f8cc41430c0 legacy=0x7f8cc41d47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:29.849 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.847+0000 7f8ccaad4640 1 -- 172.21.15.155:0/827496220 shutdown_connections 2024-03-25T18:26:29.849 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.847+0000 7f8ccaad4640 1 -- 172.21.15.155:0/827496220 >> 172.21.15.155:0/827496220 conn(0x7f8cc40b7ea0 msgr2=0x7f8cc4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:29.849 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.847+0000 7f8ccaad4640 1 -- 172.21.15.155:0/827496220 shutdown_connections 2024-03-25T18:26:29.849 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.847+0000 7f8ccaad4640 1 -- 172.21.15.155:0/827496220 wait complete. 2024-03-25T18:26:29.849 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:29.847+0000 7f8ccaad4640 1 librados: shutdown 2024-03-25T18:26:29.861 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:36.668 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:36.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.785+0000 7f8e8213d640 10 monclient: get_monmap_and_config 2024-03-25T18:26:36.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.785+0000 7f8e8213d640 10 monclient: build_initial_monmap 2024-03-25T18:26:36.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.785+0000 7f8e8213d640 10 monclient: monmap: 2024-03-25T18:26:36.786 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:26:36.786 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:26:36.786 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:26:36.786378+0000 2024-03-25T18:26:36.787 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:26:36.786378+0000 2024-03-25T18:26:36.787 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:36.787 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:26:36.787 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:26:36.787 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:26:36.787 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:26:36.787 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:26:36.788 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.786+0000 7f8e8213d640 1 Processor -- start 2024-03-25T18:26:36.788 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.786+0000 7f8e8213d640 1 -- start start 2024-03-25T18:26:36.789 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.786+0000 7f8e8213d640 10 monclient: init 2024-03-25T18:26:36.789 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.787+0000 7f8e8213d640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:36.789 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.787+0000 7f8e8213d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:26:36.789 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.787+0000 7f8e8213d640 10 monclient(hunting): picked mon.noname-b con 0x7f8e7c0a98f0 addr v1:172.21.15.155:6790/0 2024-03-25T18:26:36.789 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.787+0000 7f8e8213d640 10 monclient(hunting): picked mon.noname-a con 0x7f8e7c13f710 addr v1:172.21.15.155:6789/0 2024-03-25T18:26:36.789 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.787+0000 7f8e8213d640 10 monclient(hunting): picked mon.noname-c con 0x7f8e7c1430c0 addr v1:172.21.15.167:6789/0 2024-03-25T18:26:36.789 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.787+0000 7f8e8213d640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e7c142520 con 0x7f8e7c13f710 2024-03-25T18:26:36.789 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.787+0000 7f8e8213d640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e7c0b28a0 con 0x7f8e7c1430c0 2024-03-25T18:26:36.789 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.787+0000 7f8e8213d640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e7c0b2a30 con 0x7f8e7c0a98f0 2024-03-25T18:26:36.789 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.787+0000 7f8e8213d640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:36.789 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.787+0000 7f8e8213d640 10 monclient(hunting): authenticate will time out at 7643.097365s 2024-03-25T18:26:36.796 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.795+0000 7f8e7bfff640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f8e7c1430c0 0x7f8e7c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:60940/0 (socket says 172.21.15.155:60940) 2024-03-25T18:26:36.796 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.795+0000 7f8e7bfff640 1 -- 172.21.15.155:0/1654901347 learned_addr learned my addr 172.21.15.155:0/1654901347 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:26:36.808 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.807+0000 7f8e7a7fc640 1 -- 172.21.15.155:0/1654901347 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2101110069 0 0) 0x7f8e7c0b28a0 con 0x7f8e7c1430c0 2024-03-25T18:26:36.808 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.807+0000 7f8e7a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:26:36.808 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.807+0000 7f8e7a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:36.808 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.807+0000 7f8e7a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:36.808 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.807+0000 7f8e7a7fc640 10 monclient(hunting): my global_id is 4300 2024-03-25T18:26:36.808 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.807+0000 7f8e7a7fc640 1 -- 172.21.15.155:0/1654901347 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e64003520 con 0x7f8e7c1430c0 2024-03-25T18:26:36.809 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.807+0000 7f8e7a7fc640 1 -- 172.21.15.155:0/1654901347 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2530362244 0 0) 0x7f8e7c142520 con 0x7f8e7c13f710 2024-03-25T18:26:36.809 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.807+0000 7f8e7a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:26:36.809 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.807+0000 7f8e7a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:36.809 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.807+0000 7f8e7a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:36.809 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.807+0000 7f8e7a7fc640 10 monclient(hunting): my global_id is 4302 2024-03-25T18:26:36.809 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.807+0000 7f8e7a7fc640 1 -- 172.21.15.155:0/1654901347 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e7c0b28a0 con 0x7f8e7c13f710 2024-03-25T18:26:36.816 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.815+0000 7f8e7a7fc640 1 -- 172.21.15.155:0/1654901347 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1012125480 0 0) 0x7f8e64003520 con 0x7f8e7c1430c0 2024-03-25T18:26:36.816 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.815+0000 7f8e7a7fc640 1 -- 172.21.15.155:0/1654901347 >> v1:172.21.15.155:6790/0 conn(0x7f8e7c0a98f0 legacy=0x7f8e7c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:36.817 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.815+0000 7f8e7a7fc640 1 -- 172.21.15.155:0/1654901347 >> v1:172.21.15.155:6789/0 conn(0x7f8e7c13f710 legacy=0x7f8e7c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:36.817 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.815+0000 7f8e7a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:26:36.817 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.815+0000 7f8e7a7fc640 1 monclient: found mon.noname-c 2024-03-25T18:26:36.817 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.815+0000 7f8e7a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:36.817 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.815+0000 7f8e7a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.167:6789/0 2024-03-25T18:26:36.817 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.815+0000 7f8e7a7fc640 1 -- 172.21.15.155:0/1654901347 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e7c0b44c0 con 0x7f8e7c1430c0 2024-03-25T18:26:36.817 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.815+0000 7f8e7a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:26:36.817 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.815+0000 7f8e7a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:26:36.817 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.815+0000 7f8e7a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:36.817 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.815+0000 7f8e7a7fc640 1 -- 172.21.15.155:0/1654901347 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f8e74001a10 con 0x7f8e7c1430c0 2024-03-25T18:26:36.817 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.815+0000 7f8e8213d640 5 monclient: authenticate success, global_id 4300 2024-03-25T18:26:36.817 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.815+0000 7f8e8213d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:36.817 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.815+0000 7f8e7a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:36.818 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.815+0000 7f8e7a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:36.818 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.815+0000 7f8e7a7fc640 10 monclient: dump: 2024-03-25T18:26:36.818 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:26:36.818 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:26:36.818 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:36.818 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:36.818 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:36.818 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:26:36.818 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:26:36.818 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:26:36.818 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:26:36.818 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:26:36.818 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.815+0000 7f8e8213d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:36.827 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.826+0000 7f8e7a7fc640 1 -- 172.21.15.155:0/1654901347 <== mon.1 v1:172.21.15.167:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e74001b70 con 0x7f8e7c1430c0 2024-03-25T18:26:36.827 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.826+0000 7f8e7a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:36.827 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.826+0000 7f8e7a7fc640 1 -- 172.21.15.155:0/1654901347 <== mon.1 v1:172.21.15.167:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f8e74001f80 con 0x7f8e7c1430c0 2024-03-25T18:26:36.827 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.826+0000 7f8e7a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:36.827 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.826+0000 7f8e7a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:36.827 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.826+0000 7f8e7a7fc640 10 monclient: dump: 2024-03-25T18:26:36.827 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:26:36.827 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:26:36.827 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:36.827 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:36.828 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:36.828 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:26:36.828 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:26:36.828 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:26:36.828 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:26:36.828 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:26:36.828 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.826+0000 7f8e8213d640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:36.828 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.826+0000 7f8e8213d640 4 set_mon_vals no callback set 2024-03-25T18:26:36.829 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.827+0000 7f8e8213d640 10 monclient: shutdown 2024-03-25T18:26:36.829 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.827+0000 7f8e8213d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:36.830 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.827+0000 7f8e8213d640 1 -- 172.21.15.155:0/1654901347 >> v1:172.21.15.167:6789/0 conn(0x7f8e7c1430c0 legacy=0x7f8e7c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:36.830 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.827+0000 7f8e8213d640 1 -- 172.21.15.155:0/1654901347 shutdown_connections 2024-03-25T18:26:36.830 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.827+0000 7f8e8213d640 1 -- 172.21.15.155:0/1654901347 >> 172.21.15.155:0/1654901347 conn(0x7f8e7c0b7ea0 msgr2=0x7f8e7c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:36.830 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.827+0000 7f8e8213d640 1 -- 172.21.15.155:0/1654901347 shutdown_connections 2024-03-25T18:26:36.830 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.827+0000 7f8e8213d640 1 -- 172.21.15.155:0/1654901347 wait complete. 2024-03-25T18:26:36.830 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.828+0000 7f8e8213d640 10 monclient: build_initial_monmap 2024-03-25T18:26:36.830 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.828+0000 7f8e8213d640 10 monclient: monmap: 2024-03-25T18:26:36.830 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:26:36.830 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:36.830 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:26:36.830 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:26:36.830 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:36.830 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:26:36.831 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:26:36.831 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:26:36.831 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:26:36.831 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:26:36.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.828+0000 7f8e8213d640 1 librados: starting msgr at 2024-03-25T18:26:36.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.828+0000 7f8e8213d640 1 librados: starting objecter 2024-03-25T18:26:36.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.828+0000 7f8e8213d640 1 Processor -- start 2024-03-25T18:26:36.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.829+0000 7f8e8213d640 1 -- start start 2024-03-25T18:26:36.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.829+0000 7f8e8213d640 1 librados: setting wanted keys 2024-03-25T18:26:36.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.829+0000 7f8e8213d640 1 librados: calling monclient init 2024-03-25T18:26:36.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.829+0000 7f8e8213d640 10 monclient: init 2024-03-25T18:26:36.833 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.829+0000 7f8e8213d640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:36.833 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.829+0000 7f8e8213d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:26:36.833 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.829+0000 7f8e8213d640 10 monclient(hunting): picked mon.noname-c con 0x7f8e7c0a98f0 addr v1:172.21.15.155:6790/0 2024-03-25T18:26:36.833 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.829+0000 7f8e8213d640 10 monclient(hunting): picked mon.noname-a con 0x7f8e7c13f710 addr v1:172.21.15.155:6789/0 2024-03-25T18:26:36.833 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.829+0000 7f8e8213d640 10 monclient(hunting): picked mon.noname-b con 0x7f8e7c1430c0 addr v1:172.21.15.167:6789/0 2024-03-25T18:26:36.833 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.829+0000 7f8e8213d640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e7c137190 con 0x7f8e7c13f710 2024-03-25T18:26:36.833 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.829+0000 7f8e8213d640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e7c1d41e0 con 0x7f8e7c1430c0 2024-03-25T18:26:36.833 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.829+0000 7f8e8213d640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e7c1d4430 con 0x7f8e7c0a98f0 2024-03-25T18:26:36.833 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.829+0000 7f8e8213d640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:36.833 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.829+0000 7f8e8213d640 10 monclient(hunting): authenticate will time out at 7643.139576s 2024-03-25T18:26:36.834 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.833+0000 7f8e7affd640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f8e7c13f710 0x7f8e7c1352e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:50096/0 (socket says 172.21.15.155:50096) 2024-03-25T18:26:36.835 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.834+0000 7f8e7affd640 1 -- 172.21.15.155:0/3791024113 learned_addr learned my addr 172.21.15.155:0/3791024113 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:26:36.835 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.834+0000 7f8e63fff640 10 client.?.objecter ms_handle_connect 0x7f8e7c13f710 2024-03-25T18:26:36.835 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.834+0000 7f8e63fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:36.836 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.835+0000 7f8e7affd640 1 -- 172.21.15.155:0/3791024113 >> v1:172.21.15.155:6789/0 conn(0x7f8e7c13f710 legacy=0x7f8e7c1352e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-25T18:26:36.836 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.835+0000 7f8e7affd640 1 -- 172.21.15.155:0/3791024113 >> v1:172.21.15.155:6789/0 conn(0x7f8e7c13f710 legacy=0x7f8e7c1352e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:26:36.836 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.835+0000 7f8e7affd640 1 --1- 172.21.15.155:0/3791024113 >> v1:172.21.15.155:6789/0 conn(0x7f8e7c13f710 0x7f8e7c1352e0 :-1 s=OPENED pgs=9636 cs=1 l=1).handle_message read tag failed 2024-03-25T18:26:36.836 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.835+0000 7f8e7affd640 1 --1- 172.21.15.155:0/3791024113 >> v1:172.21.15.155:6789/0 conn(0x7f8e7c13f710 0x7f8e7c1352e0 :-1 s=OPENED pgs=9636 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:26:36.836 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.835+0000 7f8e63fff640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.155:6789/0 2024-03-25T18:26:36.842 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.842+0000 7f8e63fff640 10 client.?.objecter ms_handle_connect 0x7f8e7c1430c0 2024-03-25T18:26:36.843 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.842+0000 7f8e63fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:36.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.847+0000 7f8e63fff640 10 client.?.objecter ms_handle_connect 0x7f8e7c0a98f0 2024-03-25T18:26:36.848 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.847+0000 7f8e63fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:36.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.862+0000 7f8e63fff640 1 -- 172.21.15.155:0/3791024113 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3871146735 0 0) 0x7f8e7c1d4430 con 0x7f8e7c0a98f0 2024-03-25T18:26:36.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.862+0000 7f8e63fff640 10 monclient(hunting): _negotiate 2024-03-25T18:26:36.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.862+0000 7f8e63fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:36.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.862+0000 7f8e63fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:36.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.862+0000 7f8e63fff640 10 monclient(hunting): my global_id is 4310 2024-03-25T18:26:36.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.862+0000 7f8e63fff640 1 -- 172.21.15.155:0/3791024113 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e68004eb0 con 0x7f8e7c0a98f0 2024-03-25T18:26:36.869 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.869+0000 7f8e63fff640 1 -- 172.21.15.155:0/3791024113 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 857279133 0 0) 0x7f8e7c1d41e0 con 0x7f8e7c1430c0 2024-03-25T18:26:36.869 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.869+0000 7f8e63fff640 10 monclient(hunting): _negotiate 2024-03-25T18:26:36.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.869+0000 7f8e63fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:36.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.869+0000 7f8e63fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:36.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.869+0000 7f8e63fff640 10 monclient(hunting): my global_id is 4303 2024-03-25T18:26:36.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.869+0000 7f8e63fff640 1 -- 172.21.15.155:0/3791024113 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e7c1d4430 con 0x7f8e7c1430c0 2024-03-25T18:26:36.886 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.885+0000 7f8e63fff640 1 -- 172.21.15.155:0/3791024113 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 562306871 0 0) 0x7f8e7c1d4430 con 0x7f8e7c1430c0 2024-03-25T18:26:36.886 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.885+0000 7f8e63fff640 1 -- 172.21.15.155:0/3791024113 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e7c1d41e0 con 0x7f8e7c1430c0 2024-03-25T18:26:36.886 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.885+0000 7f8e63fff640 1 -- 172.21.15.155:0/3791024113 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f8e74001e80 con 0x7f8e7c1430c0 2024-03-25T18:26:36.887 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.885+0000 7f8e63fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:36.887 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.885+0000 7f8e63fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:36.887 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.885+0000 7f8e63fff640 10 monclient(hunting): dump: 2024-03-25T18:26:36.887 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:26:36.887 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:26:36.887 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:36.887 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:36.887 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:36.887 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:26:36.887 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:26:36.887 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:26:36.887 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:26:36.887 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:26:36.887 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.885+0000 7f8e63fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:36.895 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.894+0000 7f8e63fff640 1 -- 172.21.15.155:0/3791024113 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3718234290 0 0) 0x7f8e68004eb0 con 0x7f8e7c0a98f0 2024-03-25T18:26:36.895 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.894+0000 7f8e63fff640 1 -- 172.21.15.155:0/3791024113 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e7c1d4430 con 0x7f8e7c0a98f0 2024-03-25T18:26:36.895 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.894+0000 7f8e63fff640 1 -- 172.21.15.155:0/3791024113 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f8e6c001930 con 0x7f8e7c0a98f0 2024-03-25T18:26:36.896 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.894+0000 7f8e63fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:36.896 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.894+0000 7f8e63fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:36.896 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.894+0000 7f8e63fff640 10 monclient(hunting): dump: 2024-03-25T18:26:36.896 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:26:36.896 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:26:36.896 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:36.896 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:36.896 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:36.896 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:26:36.896 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:26:36.896 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:26:36.896 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:26:36.896 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:26:36.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.914+0000 7f8e63fff640 1 -- 172.21.15.155:0/3791024113 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3314549014 0 0) 0x7f8e7c1d4430 con 0x7f8e7c0a98f0 2024-03-25T18:26:36.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.914+0000 7f8e63fff640 1 -- 172.21.15.155:0/3791024113 >> v1:172.21.15.167:6789/0 conn(0x7f8e7c1430c0 legacy=0x7f8e7c130ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:36.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.914+0000 7f8e63fff640 1 -- 172.21.15.155:0/3791024113 >> v1:172.21.15.155:6789/0 conn(0x7f8e7c13f710 legacy=0x7f8e7c1352e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:26:36.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.914+0000 7f8e63fff640 10 monclient: _finish_hunting 0 2024-03-25T18:26:36.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.914+0000 7f8e63fff640 1 monclient: found mon.c 2024-03-25T18:26:36.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.914+0000 7f8e63fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:36.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.914+0000 7f8e63fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:26:36.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.914+0000 7f8e63fff640 1 -- 172.21.15.155:0/3791024113 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e7c13bf90 con 0x7f8e7c0a98f0 2024-03-25T18:26:36.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.914+0000 7f8e63fff640 10 monclient: _finish_auth 0 2024-03-25T18:26:36.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.914+0000 7f8e63fff640 10 monclient: _check_auth_tickets 2024-03-25T18:26:36.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.914+0000 7f8e63fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:36.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.914+0000 7f8e8213d640 5 monclient: authenticate success, global_id 4310 2024-03-25T18:26:36.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.914+0000 7f8e8213d640 10 monclient: _renew_subs 2024-03-25T18:26:36.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.914+0000 7f8e8213d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:26:36.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.914+0000 7f8e8213d640 1 -- 172.21.15.155:0/3791024113 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e7c1d5060 con 0x7f8e7c0a98f0 2024-03-25T18:26:36.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.914+0000 7f8e8213d640 10 client.4310.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:36.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.914+0000 7f8e8213d640 10 monclient: _renew_subs 2024-03-25T18:26:36.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.914+0000 7f8e8213d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:26:36.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.914+0000 7f8e8213d640 1 -- 172.21.15.155:0/3791024113 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e7c139f00 con 0x7f8e7c0a98f0 2024-03-25T18:26:36.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.914+0000 7f8e8213d640 1 librados: init done 2024-03-25T18:26:36.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.915+0000 7f8e8213d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:36.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.916+0000 7f8e8213d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:36.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.916+0000 7f8e8213d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:26:36.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:36.916+0000 7f8e8213d640 1 -- 172.21.15.155:0/3791024113 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e7c1d45c0 con 0x7f8e7c0a98f0 2024-03-25T18:26:37.227 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:37.226+0000 7f8e63fff640 1 -- 172.21.15.155:0/3791024113 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e6c0035f0 con 0x7f8e7c0a98f0 2024-03-25T18:26:37.228 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:37.226+0000 7f8e63fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:37.228 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:37.226+0000 7f8e63fff640 1 -- 172.21.15.155:0/3791024113 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f8e6c003900 con 0x7f8e7c0a98f0 2024-03-25T18:26:37.228 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:37.226+0000 7f8e63fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:37.228 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:37.226+0000 7f8e63fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:37.228 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:37.226+0000 7f8e63fff640 10 monclient: dump: 2024-03-25T18:26:37.228 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:26:37.228 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:26:37.228 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:37.228 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:37.228 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:37.228 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:26:37.228 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:26:37.228 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:26:37.229 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:26:37.229 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:26:37.229 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:37.226+0000 7f8e79ffb640 4 set_mon_vals no callback set 2024-03-25T18:26:37.229 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:37.227+0000 7f8e63fff640 1 -- 172.21.15.155:0/3791024113 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7f8e6c0198d0 con 0x7f8e7c0a98f0 2024-03-25T18:26:37.229 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:37.228+0000 7f8e7affd640 10 monclient: get_auth_request con 0x7f8e6806ddf0 auth_method 0 2024-03-25T18:26:37.229 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:37.228+0000 7f8e63fff640 10 client.4310.objecter ms_handle_connect 0x7f8e6806ddf0 2024-03-25T18:26:37.235 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:37.234+0000 7f8e63fff640 1 -- 172.21.15.155:0/3791024113 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7f8e6c0535f0 con 0x7f8e7c0a98f0 2024-03-25T18:26:37.235 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:37.234+0000 7f8e63fff640 10 client.4310.objecter ms_dispatch 0x7f8e7c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:37.235 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:37.234+0000 7f8e63fff640 3 client.4310.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:37.235 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:37.234+0000 7f8e63fff640 3 client.4310.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:37.235 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:37.234+0000 7f8e63fff640 20 client.4310.objecter dump_active .. 0 homeless 2024-03-25T18:26:37.236 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:37.234+0000 7f8e63fff640 1 -- 172.21.15.155:0/3791024113 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8e6c04ef30 con 0x7f8e7c0a98f0 2024-03-25T18:26:37.236 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:37.234+0000 7f8e63fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:37.236 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:37.234+0000 7f8e63fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:37.473 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:37.472+0000 7f8e8213d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:37.473 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:37.472+0000 7f8e8213d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:37.473 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:37.472+0000 7f8e8213d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:26:37.473 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:37.472+0000 7f8e8213d640 1 -- 172.21.15.155:0/3791024113 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8e7c0627b0 con 0x7f8e7c0a98f0 2024-03-25T18:26:37.568 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:37.567+0000 7f8e63fff640 1 -- 172.21.15.155:0/3791024113 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7f8e6c051030 con 0x7f8e7c0a98f0 2024-03-25T18:26:37.568 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:37.567+0000 7f8e63fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:37.568 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:37.567+0000 7f8e63fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:37.569 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:26:37.569 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:37.571 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:37.570+0000 7f8e8213d640 10 librados: watch_flush enter 2024-03-25T18:26:37.571 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:37.570+0000 7f8e8213d640 10 librados: watch_flush exit 2024-03-25T18:26:37.571 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:37.570+0000 7f8e8213d640 20 client.4310.objecter shutdown clearing up homeless session... 2024-03-25T18:26:37.571 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:37.570+0000 7f8e8213d640 10 client.4310.objecter successfully canceled tick 2024-03-25T18:26:37.571 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:37.570+0000 7f8e8213d640 1 -- 172.21.15.155:0/3791024113 >> v1:172.21.15.167:6800/2919070503 conn(0x7f8e6806ddf0 legacy=0x7f8e680702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:37.572 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:37.570+0000 7f8e8213d640 10 monclient: shutdown 2024-03-25T18:26:37.572 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:37.570+0000 7f8e8213d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:37.572 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:37.570+0000 7f8e8213d640 1 -- 172.21.15.155:0/3791024113 >> v1:172.21.15.155:6790/0 conn(0x7f8e7c0a98f0 legacy=0x7f8e7c1d4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:37.572 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:37.570+0000 7f8e8213d640 1 -- 172.21.15.155:0/3791024113 shutdown_connections 2024-03-25T18:26:37.572 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:37.570+0000 7f8e8213d640 1 -- 172.21.15.155:0/3791024113 >> 172.21.15.155:0/3791024113 conn(0x7f8e7c0b7ea0 msgr2=0x7f8e7c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:37.572 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:37.570+0000 7f8e8213d640 1 -- 172.21.15.155:0/3791024113 shutdown_connections 2024-03-25T18:26:37.572 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:37.570+0000 7f8e8213d640 1 -- 172.21.15.155:0/3791024113 wait complete. 2024-03-25T18:26:37.572 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:37.570+0000 7f8e8213d640 1 librados: shutdown 2024-03-25T18:26:37.584 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:40.657 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:02:14.262 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:40.667 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:02:14.272 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:40.707 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:02:14.311 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:40.707 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:02:14.312 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:40.721 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:02:14.326 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:40.733 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:02:14.312 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:40.739 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:02:14.317 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:40.752 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:02:14.357 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:40.767 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:02:14.345 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:44.392 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:44.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.510+0000 7f54effd4640 10 monclient: get_monmap_and_config 2024-03-25T18:26:44.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.510+0000 7f54effd4640 10 monclient: build_initial_monmap 2024-03-25T18:26:44.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.510+0000 7f54effd4640 10 monclient: monmap: 2024-03-25T18:26:44.511 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:26:44.511 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:26:44.511 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:26:44.511289+0000 2024-03-25T18:26:44.511 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:26:44.511289+0000 2024-03-25T18:26:44.512 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:44.512 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:26:44.512 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:26:44.512 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:26:44.512 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:26:44.512 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:26:44.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.511+0000 7f54effd4640 1 Processor -- start 2024-03-25T18:26:44.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.511+0000 7f54effd4640 1 -- start start 2024-03-25T18:26:44.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.511+0000 7f54effd4640 10 monclient: init 2024-03-25T18:26:44.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.512+0000 7f54effd4640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:44.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.512+0000 7f54effd4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:26:44.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.512+0000 7f54effd4640 10 monclient(hunting): picked mon.noname-c con 0x7f54e80a98f0 addr v1:172.21.15.167:6789/0 2024-03-25T18:26:44.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.512+0000 7f54effd4640 10 monclient(hunting): picked mon.noname-a con 0x7f54e813f710 addr v1:172.21.15.155:6789/0 2024-03-25T18:26:44.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.512+0000 7f54effd4640 10 monclient(hunting): picked mon.noname-b con 0x7f54e81430c0 addr v1:172.21.15.155:6790/0 2024-03-25T18:26:44.514 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.512+0000 7f54effd4640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54e8142520 con 0x7f54e813f710 2024-03-25T18:26:44.514 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.512+0000 7f54effd4640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54e80b28a0 con 0x7f54e80a98f0 2024-03-25T18:26:44.514 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.512+0000 7f54effd4640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54e80b2a30 con 0x7f54e81430c0 2024-03-25T18:26:44.514 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.512+0000 7f54effd4640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:44.514 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.512+0000 7f54effd4640 10 monclient(hunting): authenticate will time out at 7650.821907s 2024-03-25T18:26:44.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.518+0000 7f54ee571640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f54e81430c0 0x7f54e814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:51666/0 (socket says 172.21.15.155:51666) 2024-03-25T18:26:44.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.518+0000 7f54ee571640 1 -- 172.21.15.155:0/3759975428 learned_addr learned my addr 172.21.15.155:0/3759975428 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:26:44.534 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.533+0000 7f54ecd6e640 1 -- 172.21.15.155:0/3759975428 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4176349024 0 0) 0x7f54e80b2a30 con 0x7f54e81430c0 2024-03-25T18:26:44.534 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.533+0000 7f54ecd6e640 10 monclient(hunting): _negotiate 2024-03-25T18:26:44.534 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.533+0000 7f54ecd6e640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:44.535 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.533+0000 7f54ecd6e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:44.535 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.533+0000 7f54ecd6e640 10 monclient(hunting): my global_id is 4313 2024-03-25T18:26:44.535 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.533+0000 7f54ecd6e640 1 -- 172.21.15.155:0/3759975428 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54d4003520 con 0x7f54e81430c0 2024-03-25T18:26:44.541 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.540+0000 7f54ecd6e640 1 -- 172.21.15.155:0/3759975428 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 712998280 0 0) 0x7f54e80b28a0 con 0x7f54e80a98f0 2024-03-25T18:26:44.541 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.540+0000 7f54ecd6e640 10 monclient(hunting): _negotiate 2024-03-25T18:26:44.541 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.540+0000 7f54ecd6e640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:44.541 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.540+0000 7f54ecd6e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:44.541 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.540+0000 7f54ecd6e640 10 monclient(hunting): my global_id is 4306 2024-03-25T18:26:44.542 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.540+0000 7f54ecd6e640 1 -- 172.21.15.155:0/3759975428 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54e80b2a30 con 0x7f54e80a98f0 2024-03-25T18:26:44.543 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.543+0000 7f54ecd6e640 1 -- 172.21.15.155:0/3759975428 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 145807287 0 0) 0x7f54e8142520 con 0x7f54e813f710 2024-03-25T18:26:44.544 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.543+0000 7f54ecd6e640 10 monclient(hunting): _negotiate 2024-03-25T18:26:44.544 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.543+0000 7f54ecd6e640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:44.544 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.543+0000 7f54ecd6e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:44.544 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.543+0000 7f54ecd6e640 10 monclient(hunting): my global_id is 4308 2024-03-25T18:26:44.544 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.543+0000 7f54ecd6e640 1 -- 172.21.15.155:0/3759975428 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54e80b28a0 con 0x7f54e813f710 2024-03-25T18:26:44.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.544+0000 7f54ecd6e640 1 -- 172.21.15.155:0/3759975428 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3711887442 0 0) 0x7f54d4003520 con 0x7f54e81430c0 2024-03-25T18:26:44.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.544+0000 7f54ecd6e640 1 -- 172.21.15.155:0/3759975428 >> v1:172.21.15.167:6789/0 conn(0x7f54e80a98f0 legacy=0x7f54e80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:44.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.544+0000 7f54ecd6e640 1 -- 172.21.15.155:0/3759975428 >> v1:172.21.15.155:6789/0 conn(0x7f54e813f710 legacy=0x7f54e81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:44.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.544+0000 7f54ecd6e640 10 monclient: _finish_hunting 0 2024-03-25T18:26:44.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.544+0000 7f54ecd6e640 1 monclient: found mon.noname-b 2024-03-25T18:26:44.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.544+0000 7f54ecd6e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:44.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.544+0000 7f54ecd6e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-25T18:26:44.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.544+0000 7f54ecd6e640 1 -- 172.21.15.155:0/3759975428 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54e80b44c0 con 0x7f54e81430c0 2024-03-25T18:26:44.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.544+0000 7f54ecd6e640 10 monclient: _finish_auth 0 2024-03-25T18:26:44.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.544+0000 7f54ecd6e640 10 monclient: _check_auth_tickets 2024-03-25T18:26:44.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.544+0000 7f54ecd6e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:44.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.544+0000 7f54ecd6e640 1 -- 172.21.15.155:0/3759975428 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f54e0001a10 con 0x7f54e81430c0 2024-03-25T18:26:44.547 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.544+0000 7f54effd4640 5 monclient: authenticate success, global_id 4313 2024-03-25T18:26:44.547 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.544+0000 7f54effd4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:44.547 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.544+0000 7f54ecd6e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:44.547 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.544+0000 7f54ecd6e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:44.547 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.544+0000 7f54ecd6e640 10 monclient: dump: 2024-03-25T18:26:44.547 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:26:44.547 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:26:44.547 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:44.547 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:44.547 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:44.547 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:26:44.547 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:26:44.547 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:26:44.548 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:26:44.548 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:26:44.548 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.544+0000 7f54effd4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:44.567 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.567+0000 7f54ecd6e640 1 -- 172.21.15.155:0/3759975428 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f54e0001c30 con 0x7f54e81430c0 2024-03-25T18:26:44.567 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.567+0000 7f54ecd6e640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:44.568 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.567+0000 7f54ecd6e640 1 -- 172.21.15.155:0/3759975428 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f54e0001fc0 con 0x7f54e81430c0 2024-03-25T18:26:44.568 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.567+0000 7f54effd4640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:44.568 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.567+0000 7f54effd4640 4 set_mon_vals no callback set 2024-03-25T18:26:44.568 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.567+0000 7f54ecd6e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:44.568 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.567+0000 7f54ecd6e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:44.568 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.567+0000 7f54ecd6e640 10 monclient: dump: 2024-03-25T18:26:44.568 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:26:44.568 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:26:44.568 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:44.568 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:44.568 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:44.568 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:26:44.568 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:26:44.569 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:26:44.569 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:26:44.569 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:26:44.569 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.567+0000 7f54effd4640 10 monclient: shutdown 2024-03-25T18:26:44.569 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.567+0000 7f54effd4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:44.569 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.567+0000 7f54effd4640 1 -- 172.21.15.155:0/3759975428 >> v1:172.21.15.155:6790/0 conn(0x7f54e81430c0 legacy=0x7f54e814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:44.570 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.568+0000 7f54effd4640 1 -- 172.21.15.155:0/3759975428 shutdown_connections 2024-03-25T18:26:44.570 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.568+0000 7f54effd4640 1 -- 172.21.15.155:0/3759975428 >> 172.21.15.155:0/3759975428 conn(0x7f54e80b7ea0 msgr2=0x7f54e80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:44.570 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.568+0000 7f54effd4640 1 -- 172.21.15.155:0/3759975428 shutdown_connections 2024-03-25T18:26:44.570 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.568+0000 7f54effd4640 1 -- 172.21.15.155:0/3759975428 wait complete. 2024-03-25T18:26:44.570 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.569+0000 7f54effd4640 10 monclient: build_initial_monmap 2024-03-25T18:26:44.570 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.569+0000 7f54effd4640 10 monclient: monmap: 2024-03-25T18:26:44.570 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:26:44.570 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:44.570 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:26:44.570 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:26:44.570 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:44.570 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:26:44.570 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:26:44.571 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:26:44.571 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:26:44.571 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:26:44.572 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.569+0000 7f54effd4640 1 librados: starting msgr at 2024-03-25T18:26:44.572 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.569+0000 7f54effd4640 1 librados: starting objecter 2024-03-25T18:26:44.572 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.569+0000 7f54effd4640 1 Processor -- start 2024-03-25T18:26:44.572 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.569+0000 7f54effd4640 1 -- start start 2024-03-25T18:26:44.572 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.569+0000 7f54effd4640 1 librados: setting wanted keys 2024-03-25T18:26:44.572 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.569+0000 7f54effd4640 1 librados: calling monclient init 2024-03-25T18:26:44.572 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.569+0000 7f54effd4640 10 monclient: init 2024-03-25T18:26:44.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.570+0000 7f54effd4640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:44.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.570+0000 7f54effd4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:26:44.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.570+0000 7f54effd4640 10 monclient(hunting): picked mon.noname-a con 0x7f54e80a98f0 addr v1:172.21.15.155:6789/0 2024-03-25T18:26:44.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.570+0000 7f54effd4640 10 monclient(hunting): picked mon.noname-c con 0x7f54e813f710 addr v1:172.21.15.155:6790/0 2024-03-25T18:26:44.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.570+0000 7f54effd4640 10 monclient(hunting): picked mon.noname-b con 0x7f54e81430c0 addr v1:172.21.15.167:6789/0 2024-03-25T18:26:44.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.570+0000 7f54effd4640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54e8138b40 con 0x7f54e80a98f0 2024-03-25T18:26:44.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.570+0000 7f54effd4640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54e8138e20 con 0x7f54e81430c0 2024-03-25T18:26:44.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.570+0000 7f54effd4640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54e8139100 con 0x7f54e813f710 2024-03-25T18:26:44.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.570+0000 7f54effd4640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:44.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.570+0000 7f54effd4640 10 monclient(hunting): authenticate will time out at 7650.880328s 2024-03-25T18:26:44.579 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.578+0000 7f54ee571640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f54e81430c0 0x7f54e81e1fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:37464/0 (socket says 172.21.15.155:37464) 2024-03-25T18:26:44.579 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.578+0000 7f54ee571640 1 -- 172.21.15.155:0/3547823586 learned_addr learned my addr 172.21.15.155:0/3547823586 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:26:44.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.581+0000 7f54de7fc640 10 client.?.objecter ms_handle_connect 0x7f54e81430c0 2024-03-25T18:26:44.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.581+0000 7f54de7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:44.583 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.582+0000 7f54de7fc640 10 client.?.objecter ms_handle_connect 0x7f54e80a98f0 2024-03-25T18:26:44.583 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.582+0000 7f54de7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:44.591 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.590+0000 7f54de7fc640 1 -- 172.21.15.155:0/3547823586 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3983757849 0 0) 0x7f54e8138e20 con 0x7f54e81430c0 2024-03-25T18:26:44.591 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.590+0000 7f54de7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:26:44.591 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.590+0000 7f54de7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:44.591 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.590+0000 7f54de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:44.591 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.590+0000 7f54de7fc640 10 monclient(hunting): my global_id is 4309 2024-03-25T18:26:44.591 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.590+0000 7f54de7fc640 1 -- 172.21.15.155:0/3547823586 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54cc004eb0 con 0x7f54e81430c0 2024-03-25T18:26:44.598 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.597+0000 7f54de7fc640 1 -- 172.21.15.155:0/3547823586 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 932566272 0 0) 0x7f54e8138b40 con 0x7f54e80a98f0 2024-03-25T18:26:44.598 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.597+0000 7f54de7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:26:44.598 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.597+0000 7f54de7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:44.598 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.597+0000 7f54de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:44.598 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.597+0000 7f54de7fc640 10 monclient(hunting): my global_id is 4311 2024-03-25T18:26:44.598 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.597+0000 7f54de7fc640 1 -- 172.21.15.155:0/3547823586 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54e8138e20 con 0x7f54e80a98f0 2024-03-25T18:26:44.605 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.604+0000 7f54de7fc640 1 -- 172.21.15.155:0/3547823586 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4235522197 0 0) 0x7f54cc004eb0 con 0x7f54e81430c0 2024-03-25T18:26:44.605 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.604+0000 7f54de7fc640 1 -- 172.21.15.155:0/3547823586 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54e8138b40 con 0x7f54e81430c0 2024-03-25T18:26:44.605 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.604+0000 7f54de7fc640 1 -- 172.21.15.155:0/3547823586 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f54e0001f40 con 0x7f54e81430c0 2024-03-25T18:26:44.605 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.604+0000 7f54de7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:44.606 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.604+0000 7f54de7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:44.606 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.604+0000 7f54de7fc640 10 monclient(hunting): dump: 2024-03-25T18:26:44.606 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:26:44.606 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:26:44.606 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:44.606 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:44.606 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:44.606 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:26:44.606 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:26:44.606 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:26:44.606 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:26:44.606 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:26:44.606 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.604+0000 7f54de7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:44.608 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.608+0000 7f54de7fc640 10 client.?.objecter ms_handle_connect 0x7f54e813f710 2024-03-25T18:26:44.609 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.608+0000 7f54de7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:44.610 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.609+0000 7f54de7fc640 1 -- 172.21.15.155:0/3547823586 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2448421478 0 0) 0x7f54e8138e20 con 0x7f54e80a98f0 2024-03-25T18:26:44.610 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.609+0000 7f54de7fc640 1 -- 172.21.15.155:0/3547823586 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54cc004eb0 con 0x7f54e80a98f0 2024-03-25T18:26:44.611 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.609+0000 7f54de7fc640 1 -- 172.21.15.155:0/3547823586 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f54e4001970 con 0x7f54e80a98f0 2024-03-25T18:26:44.611 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.609+0000 7f54de7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:44.611 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.609+0000 7f54de7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:44.611 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.609+0000 7f54de7fc640 10 monclient(hunting): dump: 2024-03-25T18:26:44.611 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:26:44.611 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:26:44.611 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:44.611 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:44.611 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:44.611 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:26:44.611 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:26:44.611 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:26:44.611 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:26:44.611 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:26:44.615 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.614+0000 7f54de7fc640 1 -- 172.21.15.155:0/3547823586 <== mon.1 v1:172.21.15.167:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4194265171 0 0) 0x7f54e8138b40 con 0x7f54e81430c0 2024-03-25T18:26:44.615 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.614+0000 7f54de7fc640 1 -- 172.21.15.155:0/3547823586 >> v1:172.21.15.155:6790/0 conn(0x7f54e813f710 legacy=0x7f54e8137a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:44.615 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.614+0000 7f54de7fc640 1 -- 172.21.15.155:0/3547823586 >> v1:172.21.15.155:6789/0 conn(0x7f54e80a98f0 legacy=0x7f54e8137380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:44.615 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.614+0000 7f54de7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:26:44.615 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.614+0000 7f54de7fc640 1 monclient: found mon.b 2024-03-25T18:26:44.615 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.614+0000 7f54de7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:44.616 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.614+0000 7f54de7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:26:44.616 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.614+0000 7f54de7fc640 1 -- 172.21.15.155:0/3547823586 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54e81397a0 con 0x7f54e81430c0 2024-03-25T18:26:44.616 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.614+0000 7f54de7fc640 10 monclient: _finish_auth 0 2024-03-25T18:26:44.616 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.614+0000 7f54de7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:26:44.616 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.614+0000 7f54de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:44.616 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.614+0000 7f54effd4640 5 monclient: authenticate success, global_id 4309 2024-03-25T18:26:44.616 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.614+0000 7f54effd4640 10 monclient: _renew_subs 2024-03-25T18:26:44.616 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.614+0000 7f54effd4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:26:44.616 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.614+0000 7f54effd4640 1 -- 172.21.15.155:0/3547823586 --> v1:172.21.15.167:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54e8136aa0 con 0x7f54e81430c0 2024-03-25T18:26:44.616 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.614+0000 7f54effd4640 10 client.4309.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:44.616 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.614+0000 7f54effd4640 10 monclient: _renew_subs 2024-03-25T18:26:44.616 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.614+0000 7f54effd4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:26:44.616 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.614+0000 7f54effd4640 1 -- 172.21.15.155:0/3547823586 --> v1:172.21.15.167:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f54e813c320 con 0x7f54e81430c0 2024-03-25T18:26:44.616 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.614+0000 7f54effd4640 1 librados: init done 2024-03-25T18:26:44.617 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.615+0000 7f54effd4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:44.617 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.615+0000 7f54effd4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:44.617 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.615+0000 7f54effd4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:26:44.617 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.615+0000 7f54effd4640 1 -- 172.21.15.155:0/3547823586 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54e81395b0 con 0x7f54e81430c0 2024-03-25T18:26:44.897 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.896+0000 7f54de7fc640 1 -- 172.21.15.155:0/3547823586 <== mon.1 v1:172.21.15.167:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f54e00014f0 con 0x7f54e81430c0 2024-03-25T18:26:44.897 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.896+0000 7f54de7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:44.897 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.896+0000 7f54de7fc640 1 -- 172.21.15.155:0/3547823586 <== mon.1 v1:172.21.15.167:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f54e0002100 con 0x7f54e81430c0 2024-03-25T18:26:44.898 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.896+0000 7f54de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:44.898 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.896+0000 7f54de7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:44.898 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.896+0000 7f54de7fc640 10 monclient: dump: 2024-03-25T18:26:44.898 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:26:44.898 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:26:44.898 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:44.898 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:44.898 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:44.898 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:26:44.898 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:26:44.898 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:26:44.898 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:26:44.898 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:26:44.899 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.896+0000 7f54dffff640 4 set_mon_vals no callback set 2024-03-25T18:26:44.899 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.897+0000 7f54de7fc640 1 -- 172.21.15.155:0/3547823586 <== mon.1 v1:172.21.15.167:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7f54e001aea0 con 0x7f54e81430c0 2024-03-25T18:26:44.899 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.898+0000 7f54edd70640 10 monclient: get_auth_request con 0x7f54cc06de80 auth_method 0 2024-03-25T18:26:44.899 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.898+0000 7f54de7fc640 10 client.4309.objecter ms_handle_connect 0x7f54cc06de80 2024-03-25T18:26:44.906 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.905+0000 7f54de7fc640 1 -- 172.21.15.155:0/3547823586 <== mon.1 v1:172.21.15.167:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7f54e0054bc0 con 0x7f54e81430c0 2024-03-25T18:26:44.906 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.905+0000 7f54de7fc640 10 client.4309.objecter ms_dispatch 0x7f54e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:44.906 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.905+0000 7f54de7fc640 3 client.4309.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:44.906 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.905+0000 7f54de7fc640 3 client.4309.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:44.906 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.905+0000 7f54de7fc640 20 client.4309.objecter dump_active .. 0 homeless 2024-03-25T18:26:44.906 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.905+0000 7f54de7fc640 1 -- 172.21.15.155:0/3547823586 <== mon.1 v1:172.21.15.167:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f54e0050500 con 0x7f54e81430c0 2024-03-25T18:26:44.906 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.905+0000 7f54de7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:44.906 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:44.905+0000 7f54de7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:45.154 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:45.152+0000 7f54effd4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:45.154 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:45.152+0000 7f54effd4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:45.154 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:45.153+0000 7f54effd4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:26:45.154 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:45.153+0000 7f54effd4640 1 -- 172.21.15.155:0/3547823586 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f54e8139930 con 0x7f54e81430c0 2024-03-25T18:26:45.178 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:45.177+0000 7f54de7fc640 1 -- 172.21.15.155:0/3547823586 <== mon.1 v1:172.21.15.167:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7f54e0052600 con 0x7f54e81430c0 2024-03-25T18:26:45.178 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:45.177+0000 7f54de7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:45.178 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:45.177+0000 7f54de7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:45.178 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:26:45.178 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:45.180 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:45.179+0000 7f54effd4640 10 librados: watch_flush enter 2024-03-25T18:26:45.180 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:45.179+0000 7f54effd4640 10 librados: watch_flush exit 2024-03-25T18:26:45.181 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:45.179+0000 7f54effd4640 20 client.4309.objecter shutdown clearing up homeless session... 2024-03-25T18:26:45.181 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:45.179+0000 7f54effd4640 10 client.4309.objecter successfully canceled tick 2024-03-25T18:26:45.181 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:45.179+0000 7f54effd4640 1 -- 172.21.15.155:0/3547823586 >> v1:172.21.15.167:6800/2919070503 conn(0x7f54cc06de80 legacy=0x7f54cc070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:45.181 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:45.179+0000 7f54effd4640 10 monclient: shutdown 2024-03-25T18:26:45.181 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:45.179+0000 7f54effd4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:45.181 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:45.179+0000 7f54effd4640 1 -- 172.21.15.155:0/3547823586 >> v1:172.21.15.167:6789/0 conn(0x7f54e81430c0 legacy=0x7f54e81e1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:45.181 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:45.179+0000 7f54effd4640 1 -- 172.21.15.155:0/3547823586 shutdown_connections 2024-03-25T18:26:45.181 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:45.179+0000 7f54effd4640 1 -- 172.21.15.155:0/3547823586 >> 172.21.15.155:0/3547823586 conn(0x7f54e80b7ea0 msgr2=0x7f54e814ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:45.181 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:45.179+0000 7f54effd4640 1 -- 172.21.15.155:0/3547823586 shutdown_connections 2024-03-25T18:26:45.181 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:45.179+0000 7f54effd4640 1 -- 172.21.15.155:0/3547823586 wait complete. 2024-03-25T18:26:45.181 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:45.179+0000 7f54effd4640 1 librados: shutdown 2024-03-25T18:26:45.193 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:51.999 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:52.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.117+0000 7fb1a45d0640 10 monclient: get_monmap_and_config 2024-03-25T18:26:52.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.117+0000 7fb1a45d0640 10 monclient: build_initial_monmap 2024-03-25T18:26:52.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.117+0000 7fb1a45d0640 10 monclient: monmap: 2024-03-25T18:26:52.119 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:26:52.119 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:26:52.119 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:26:52.118866+0000 2024-03-25T18:26:52.119 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:26:52.118866+0000 2024-03-25T18:26:52.119 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:52.119 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:26:52.120 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:26:52.120 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:26:52.120 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:26:52.120 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:26:52.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.118+0000 7fb1a45d0640 1 Processor -- start 2024-03-25T18:26:52.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.118+0000 7fb1a45d0640 1 -- start start 2024-03-25T18:26:52.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.118+0000 7fb1a45d0640 10 monclient: init 2024-03-25T18:26:52.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.119+0000 7fb1a45d0640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:52.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.119+0000 7fb1a45d0640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:26:52.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.119+0000 7fb1a45d0640 10 monclient(hunting): picked mon.noname-b con 0x7fb19c0a98f0 addr v1:172.21.15.155:6790/0 2024-03-25T18:26:52.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.119+0000 7fb1a45d0640 10 monclient(hunting): picked mon.noname-c con 0x7fb19c13f710 addr v1:172.21.15.167:6789/0 2024-03-25T18:26:52.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.119+0000 7fb1a45d0640 10 monclient(hunting): picked mon.noname-a con 0x7fb19c1430c0 addr v1:172.21.15.155:6789/0 2024-03-25T18:26:52.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.119+0000 7fb1a45d0640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb19c142520 con 0x7fb19c1430c0 2024-03-25T18:26:52.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.119+0000 7fb1a45d0640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb19c0b28a0 con 0x7fb19c13f710 2024-03-25T18:26:52.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.119+0000 7fb1a45d0640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb19c0b2a30 con 0x7fb19c0a98f0 2024-03-25T18:26:52.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.119+0000 7fb1a45d0640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:52.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.119+0000 7fb1a45d0640 10 monclient(hunting): authenticate will time out at 7658.429574s 2024-03-25T18:26:52.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.130+0000 7fb1a1b6b640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7fb19c13f710 0x7fb19c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:46476/0 (socket says 172.21.15.155:46476) 2024-03-25T18:26:52.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.130+0000 7fb1a1b6b640 1 -- 172.21.15.155:0/2677788445 learned_addr learned my addr 172.21.15.155:0/2677788445 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:26:52.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.138+0000 7fb1a136a640 1 -- 172.21.15.155:0/2677788445 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2970243636 0 0) 0x7fb19c0b28a0 con 0x7fb19c13f710 2024-03-25T18:26:52.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.138+0000 7fb1a136a640 10 monclient(hunting): _negotiate 2024-03-25T18:26:52.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.138+0000 7fb1a136a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:52.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.138+0000 7fb1a136a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:52.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.138+0000 7fb1a136a640 10 monclient(hunting): my global_id is 4312 2024-03-25T18:26:52.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.138+0000 7fb1a136a640 1 -- 172.21.15.155:0/2677788445 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb188003520 con 0x7fb19c13f710 2024-03-25T18:26:52.149 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.148+0000 7fb1a136a640 1 -- 172.21.15.155:0/2677788445 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3845799293 0 0) 0x7fb188003520 con 0x7fb19c13f710 2024-03-25T18:26:52.149 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.148+0000 7fb1a136a640 1 -- 172.21.15.155:0/2677788445 >> v1:172.21.15.155:6790/0 conn(0x7fb19c0a98f0 legacy=0x7fb19c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:52.149 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.148+0000 7fb1a136a640 1 -- 172.21.15.155:0/2677788445 >> v1:172.21.15.155:6789/0 conn(0x7fb19c1430c0 legacy=0x7fb19c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:52.149 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.148+0000 7fb1a136a640 10 monclient: _finish_hunting 0 2024-03-25T18:26:52.150 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.148+0000 7fb1a136a640 1 monclient: found mon.noname-c 2024-03-25T18:26:52.150 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.148+0000 7fb1a136a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:52.150 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.148+0000 7fb1a136a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.167:6789/0 2024-03-25T18:26:52.150 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.148+0000 7fb1a136a640 1 -- 172.21.15.155:0/2677788445 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb19c0b44c0 con 0x7fb19c13f710 2024-03-25T18:26:52.150 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.148+0000 7fb1a136a640 10 monclient: _finish_auth 0 2024-03-25T18:26:52.150 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.148+0000 7fb1a136a640 10 monclient: _check_auth_tickets 2024-03-25T18:26:52.150 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.148+0000 7fb1a136a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:52.150 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.148+0000 7fb1a136a640 1 -- 172.21.15.155:0/2677788445 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fb18c001690 con 0x7fb19c13f710 2024-03-25T18:26:52.150 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.148+0000 7fb1a136a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:52.150 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.148+0000 7fb1a136a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:52.150 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.148+0000 7fb1a136a640 10 monclient: dump: 2024-03-25T18:26:52.150 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:26:52.150 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:26:52.151 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:52.151 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:52.151 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:52.151 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:26:52.151 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:26:52.151 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:26:52.151 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:26:52.151 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:26:52.151 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.148+0000 7fb1a45d0640 5 monclient: authenticate success, global_id 4312 2024-03-25T18:26:52.151 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.148+0000 7fb1a45d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:52.165 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.164+0000 7fb1a136a640 1 -- 172.21.15.155:0/2677788445 <== mon.1 v1:172.21.15.167:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb18c0017f0 con 0x7fb19c13f710 2024-03-25T18:26:52.165 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.164+0000 7fb1a136a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:52.165 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.164+0000 7fb1a136a640 1 -- 172.21.15.155:0/2677788445 <== mon.1 v1:172.21.15.167:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fb18c001c00 con 0x7fb19c13f710 2024-03-25T18:26:52.165 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.164+0000 7fb1a45d0640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:52.165 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.164+0000 7fb1a45d0640 4 set_mon_vals no callback set 2024-03-25T18:26:52.166 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.164+0000 7fb1a136a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:52.166 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.164+0000 7fb1a136a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:52.166 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.164+0000 7fb1a136a640 10 monclient: dump: 2024-03-25T18:26:52.166 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:26:52.166 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:26:52.166 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:52.166 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:52.166 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:52.166 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:26:52.166 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:26:52.166 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:26:52.166 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:26:52.166 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:26:52.168 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.165+0000 7fb1a45d0640 10 monclient: shutdown 2024-03-25T18:26:52.168 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.165+0000 7fb1a45d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:52.168 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.165+0000 7fb1a45d0640 1 -- 172.21.15.155:0/2677788445 >> v1:172.21.15.167:6789/0 conn(0x7fb19c13f710 legacy=0x7fb19c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:52.169 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.165+0000 7fb1a45d0640 1 -- 172.21.15.155:0/2677788445 shutdown_connections 2024-03-25T18:26:52.169 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.165+0000 7fb1a45d0640 1 -- 172.21.15.155:0/2677788445 >> 172.21.15.155:0/2677788445 conn(0x7fb19c0b7ea0 msgr2=0x7fb19c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:52.169 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.165+0000 7fb1a45d0640 1 -- 172.21.15.155:0/2677788445 shutdown_connections 2024-03-25T18:26:52.169 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.166+0000 7fb1a45d0640 1 -- 172.21.15.155:0/2677788445 wait complete. 2024-03-25T18:26:52.169 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.166+0000 7fb1a45d0640 10 monclient: build_initial_monmap 2024-03-25T18:26:52.169 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.166+0000 7fb1a45d0640 10 monclient: monmap: 2024-03-25T18:26:52.169 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:26:52.169 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:52.169 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:26:52.169 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:26:52.169 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:52.169 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:26:52.169 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:26:52.169 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:26:52.169 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:26:52.170 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:26:52.170 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.166+0000 7fb1a45d0640 1 librados: starting msgr at 2024-03-25T18:26:52.170 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.166+0000 7fb1a45d0640 1 librados: starting objecter 2024-03-25T18:26:52.170 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.167+0000 7fb1a45d0640 1 Processor -- start 2024-03-25T18:26:52.170 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.167+0000 7fb1a45d0640 1 -- start start 2024-03-25T18:26:52.170 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.167+0000 7fb1a45d0640 1 librados: setting wanted keys 2024-03-25T18:26:52.170 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.167+0000 7fb1a45d0640 1 librados: calling monclient init 2024-03-25T18:26:52.170 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.167+0000 7fb1a45d0640 10 monclient: init 2024-03-25T18:26:52.170 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.167+0000 7fb1a45d0640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:52.170 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.167+0000 7fb1a45d0640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:26:52.170 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.167+0000 7fb1a45d0640 10 monclient(hunting): picked mon.noname-b con 0x7fb19c0a98f0 addr v1:172.21.15.167:6789/0 2024-03-25T18:26:52.170 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.167+0000 7fb1a45d0640 10 monclient(hunting): picked mon.noname-c con 0x7fb19c13f710 addr v1:172.21.15.155:6790/0 2024-03-25T18:26:52.170 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.167+0000 7fb1a45d0640 10 monclient(hunting): picked mon.noname-a con 0x7fb19c1430c0 addr v1:172.21.15.155:6789/0 2024-03-25T18:26:52.171 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.167+0000 7fb1a45d0640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb19c13be70 con 0x7fb19c1430c0 2024-03-25T18:26:52.171 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.167+0000 7fb1a45d0640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb19c131660 con 0x7fb19c0a98f0 2024-03-25T18:26:52.171 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.167+0000 7fb1a45d0640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb19c1317f0 con 0x7fb19c13f710 2024-03-25T18:26:52.171 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.167+0000 7fb1a45d0640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:52.171 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.167+0000 7fb1a45d0640 10 monclient(hunting): authenticate will time out at 7658.477600s 2024-03-25T18:26:52.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.171+0000 7fb1a236c640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7fb19c0a98f0 0x7fb19c130f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:46498/0 (socket says 172.21.15.155:46498) 2024-03-25T18:26:52.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.171+0000 7fb1a236c640 1 -- 172.21.15.155:0/3050032764 learned_addr learned my addr 172.21.15.155:0/3050032764 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:26:52.173 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.172+0000 7fb192ffd640 10 client.?.objecter ms_handle_connect 0x7fb19c0a98f0 2024-03-25T18:26:52.173 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.172+0000 7fb192ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:52.187 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.186+0000 7fb192ffd640 10 client.?.objecter ms_handle_connect 0x7fb19c13f710 2024-03-25T18:26:52.187 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.186+0000 7fb192ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:52.189 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.188+0000 7fb192ffd640 10 client.?.objecter ms_handle_connect 0x7fb19c1430c0 2024-03-25T18:26:52.189 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.188+0000 7fb192ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:52.193 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.192+0000 7fb192ffd640 1 -- 172.21.15.155:0/3050032764 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2660746012 0 0) 0x7fb19c131660 con 0x7fb19c0a98f0 2024-03-25T18:26:52.193 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.192+0000 7fb192ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:52.193 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.192+0000 7fb192ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:52.193 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.192+0000 7fb192ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:52.193 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.192+0000 7fb192ffd640 10 monclient(hunting): my global_id is 4315 2024-03-25T18:26:52.193 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.192+0000 7fb192ffd640 1 -- 172.21.15.155:0/3050032764 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb180004eb0 con 0x7fb19c0a98f0 2024-03-25T18:26:52.205 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.204+0000 7fb192ffd640 1 -- 172.21.15.155:0/3050032764 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3861880197 0 0) 0x7fb19c13be70 con 0x7fb19c1430c0 2024-03-25T18:26:52.205 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.204+0000 7fb192ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:52.206 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.204+0000 7fb192ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:52.206 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.204+0000 7fb192ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:52.206 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.205+0000 7fb192ffd640 10 monclient(hunting): my global_id is 4317 2024-03-25T18:26:52.206 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.205+0000 7fb192ffd640 1 -- 172.21.15.155:0/3050032764 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb19c131660 con 0x7fb19c1430c0 2024-03-25T18:26:52.209 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.208+0000 7fb192ffd640 1 -- 172.21.15.155:0/3050032764 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2408154819 0 0) 0x7fb19c1317f0 con 0x7fb19c13f710 2024-03-25T18:26:52.209 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.208+0000 7fb192ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:52.210 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.208+0000 7fb192ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:52.210 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.208+0000 7fb192ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:52.210 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.208+0000 7fb192ffd640 10 monclient(hunting): my global_id is 4322 2024-03-25T18:26:52.210 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.208+0000 7fb192ffd640 1 -- 172.21.15.155:0/3050032764 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb19c13be70 con 0x7fb19c13f710 2024-03-25T18:26:52.220 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.219+0000 7fb192ffd640 1 -- 172.21.15.155:0/3050032764 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2575786241 0 0) 0x7fb19c131660 con 0x7fb19c1430c0 2024-03-25T18:26:52.220 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.219+0000 7fb192ffd640 1 -- 172.21.15.155:0/3050032764 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb19c1317f0 con 0x7fb19c1430c0 2024-03-25T18:26:52.220 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.219+0000 7fb192ffd640 1 -- 172.21.15.155:0/3050032764 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fb194001b00 con 0x7fb19c1430c0 2024-03-25T18:26:52.220 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.219+0000 7fb192ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:52.220 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.219+0000 7fb192ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:26:52.220 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.219+0000 7fb192ffd640 10 monclient(hunting): dump: 2024-03-25T18:26:52.221 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:26:52.221 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:26:52.221 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:52.221 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:52.221 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:52.221 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:26:52.221 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:26:52.221 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:26:52.221 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:26:52.221 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:26:52.221 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.219+0000 7fb192ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:52.235 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.233+0000 7fb192ffd640 1 -- 172.21.15.155:0/3050032764 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1887313683 0 0) 0x7fb19c1317f0 con 0x7fb19c1430c0 2024-03-25T18:26:52.235 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.233+0000 7fb192ffd640 1 -- 172.21.15.155:0/3050032764 >> v1:172.21.15.155:6790/0 conn(0x7fb19c13f710 legacy=0x7fb19c1d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:52.235 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.233+0000 7fb192ffd640 1 -- 172.21.15.155:0/3050032764 >> v1:172.21.15.167:6789/0 conn(0x7fb19c0a98f0 legacy=0x7fb19c130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:52.235 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.233+0000 7fb192ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:26:52.235 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.233+0000 7fb192ffd640 1 monclient: found mon.a 2024-03-25T18:26:52.235 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.233+0000 7fb192ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:52.235 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.233+0000 7fb192ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:26:52.235 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.233+0000 7fb192ffd640 1 -- 172.21.15.155:0/3050032764 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb19c131980 con 0x7fb19c1430c0 2024-03-25T18:26:52.235 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.233+0000 7fb192ffd640 10 monclient: _finish_auth 0 2024-03-25T18:26:52.235 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.233+0000 7fb192ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:26:52.236 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.233+0000 7fb192ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:52.236 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.233+0000 7fb1a45d0640 5 monclient: authenticate success, global_id 4317 2024-03-25T18:26:52.236 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.233+0000 7fb1a45d0640 10 monclient: _renew_subs 2024-03-25T18:26:52.236 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.233+0000 7fb1a45d0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:26:52.236 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.233+0000 7fb1a45d0640 1 -- 172.21.15.155:0/3050032764 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb19c135c60 con 0x7fb19c1430c0 2024-03-25T18:26:52.236 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.233+0000 7fb1a45d0640 10 client.4317.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:52.236 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.233+0000 7fb1a45d0640 10 monclient: _renew_subs 2024-03-25T18:26:52.236 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.233+0000 7fb1a45d0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:26:52.236 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.233+0000 7fb1a45d0640 1 -- 172.21.15.155:0/3050032764 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb19c0b7770 con 0x7fb19c1430c0 2024-03-25T18:26:52.236 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.233+0000 7fb1a45d0640 1 librados: init done 2024-03-25T18:26:52.237 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.234+0000 7fb1a45d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:52.237 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.234+0000 7fb1a45d0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:52.237 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.234+0000 7fb1a45d0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:26:52.237 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.234+0000 7fb1a45d0640 1 -- 172.21.15.155:0/3050032764 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb19c1371d0 con 0x7fb19c1430c0 2024-03-25T18:26:52.447 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.446+0000 7fb192ffd640 1 -- 172.21.15.155:0/3050032764 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb194001fb0 con 0x7fb19c1430c0 2024-03-25T18:26:52.447 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.446+0000 7fb192ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:52.447 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.446+0000 7fb1a0b69640 4 set_mon_vals no callback set 2024-03-25T18:26:52.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.485+0000 7fb192ffd640 1 -- 172.21.15.155:0/3050032764 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fb194003b10 con 0x7fb19c1430c0 2024-03-25T18:26:52.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.485+0000 7fb192ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:52.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.485+0000 7fb192ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:52.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.485+0000 7fb192ffd640 10 monclient: dump: 2024-03-25T18:26:52.487 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:26:52.487 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:26:52.487 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:52.487 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:52.487 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:52.487 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:26:52.487 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:26:52.487 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:26:52.487 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:26:52.487 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:26:52.488 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.486+0000 7fb192ffd640 1 -- 172.21.15.155:0/3050032764 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7fb194003cf0 con 0x7fb19c1430c0 2024-03-25T18:26:52.488 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.486+0000 7fb1a236c640 10 monclient: get_auth_request con 0x7fb18006de60 auth_method 0 2024-03-25T18:26:52.488 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.487+0000 7fb192ffd640 10 client.4317.objecter ms_handle_connect 0x7fb18006de60 2024-03-25T18:26:52.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.490+0000 7fb192ffd640 1 -- 172.21.15.155:0/3050032764 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7fb194053810 con 0x7fb19c1430c0 2024-03-25T18:26:52.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.490+0000 7fb192ffd640 10 client.4317.objecter ms_dispatch 0x7fb19c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:52.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.490+0000 7fb192ffd640 3 client.4317.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:52.492 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.490+0000 7fb192ffd640 3 client.4317.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:52.492 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.490+0000 7fb192ffd640 20 client.4317.objecter dump_active .. 0 homeless 2024-03-25T18:26:52.492 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.490+0000 7fb192ffd640 1 -- 172.21.15.155:0/3050032764 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb19404f320 con 0x7fb19c1430c0 2024-03-25T18:26:52.492 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.490+0000 7fb192ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:52.492 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.490+0000 7fb192ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:52.713 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.712+0000 7fb1a45d0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:52.713 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.712+0000 7fb1a45d0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:52.714 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.712+0000 7fb1a45d0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:26:52.714 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.712+0000 7fb1a45d0640 1 -- 172.21.15.155:0/3050032764 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb19c0627b0 con 0x7fb19c1430c0 2024-03-25T18:26:52.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.736+0000 7fb192ffd640 1 -- 172.21.15.155:0/3050032764 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7fb194051250 con 0x7fb19c1430c0 2024-03-25T18:26:52.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.736+0000 7fb192ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:52.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.736+0000 7fb192ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:52.738 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:26:52.738 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:52.740 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.739+0000 7fb1a45d0640 10 librados: watch_flush enter 2024-03-25T18:26:52.740 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.739+0000 7fb1a45d0640 10 librados: watch_flush exit 2024-03-25T18:26:52.740 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.739+0000 7fb1a45d0640 20 client.4317.objecter shutdown clearing up homeless session... 2024-03-25T18:26:52.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.739+0000 7fb1a45d0640 10 client.4317.objecter successfully canceled tick 2024-03-25T18:26:52.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.739+0000 7fb1a45d0640 1 -- 172.21.15.155:0/3050032764 >> v1:172.21.15.167:6800/2919070503 conn(0x7fb18006de60 legacy=0x7fb180070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:52.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.739+0000 7fb1a45d0640 10 monclient: shutdown 2024-03-25T18:26:52.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.739+0000 7fb1a45d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:52.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.739+0000 7fb1a45d0640 1 -- 172.21.15.155:0/3050032764 >> v1:172.21.15.155:6789/0 conn(0x7fb19c1430c0 legacy=0x7fb19c1e2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:52.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.739+0000 7fb1a45d0640 1 -- 172.21.15.155:0/3050032764 shutdown_connections 2024-03-25T18:26:52.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.739+0000 7fb1a45d0640 1 -- 172.21.15.155:0/3050032764 >> 172.21.15.155:0/3050032764 conn(0x7fb19c0b7ea0 msgr2=0x7fb19c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:52.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.739+0000 7fb1a45d0640 1 -- 172.21.15.155:0/3050032764 shutdown_connections 2024-03-25T18:26:52.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.739+0000 7fb1a45d0640 1 -- 172.21.15.155:0/3050032764 wait complete. 2024-03-25T18:26:52.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:52.739+0000 7fb1a45d0640 1 librados: shutdown 2024-03-25T18:26:52.751 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:59.559 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:59.674 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.673+0000 7fac4b923640 10 monclient: get_monmap_and_config 2024-03-25T18:26:59.674 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.673+0000 7fac4b923640 10 monclient: build_initial_monmap 2024-03-25T18:26:59.674 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.673+0000 7fac4b923640 10 monclient: monmap: 2024-03-25T18:26:59.675 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:26:59.675 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:26:59.675 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:26:59.674193+0000 2024-03-25T18:26:59.675 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:26:59.674193+0000 2024-03-25T18:26:59.675 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:59.675 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:26:59.676 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:26:59.676 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:26:59.676 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:26:59.676 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:26:59.679 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.673+0000 7fac4b923640 1 Processor -- start 2024-03-25T18:26:59.679 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.674+0000 7fac4b923640 1 -- start start 2024-03-25T18:26:59.679 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.674+0000 7fac4b923640 10 monclient: init 2024-03-25T18:26:59.679 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.674+0000 7fac4b923640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:59.679 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.674+0000 7fac4b923640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:26:59.680 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.674+0000 7fac4b923640 10 monclient(hunting): picked mon.noname-c con 0x7fac4406ab80 addr v1:172.21.15.167:6789/0 2024-03-25T18:26:59.680 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.674+0000 7fac4b923640 10 monclient(hunting): picked mon.noname-a con 0x7fac4406af60 addr v1:172.21.15.155:6789/0 2024-03-25T18:26:59.680 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.674+0000 7fac4b923640 10 monclient(hunting): picked mon.noname-b con 0x7fac440b00b0 addr v1:172.21.15.155:6790/0 2024-03-25T18:26:59.680 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.674+0000 7fac4b923640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac440b0500 con 0x7fac4406af60 2024-03-25T18:26:59.680 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.674+0000 7fac4b923640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac4413ffa0 con 0x7fac4406ab80 2024-03-25T18:26:59.680 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.674+0000 7fac4b923640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac440af520 con 0x7fac440b00b0 2024-03-25T18:26:59.680 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.674+0000 7fac4b923640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:59.681 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.674+0000 7fac4b923640 10 monclient(hunting): authenticate will time out at 7665.984621s 2024-03-25T18:26:59.681 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.680+0000 7fac49ec0640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fac440b00b0 0x7fac4414dcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:48200/0 (socket says 172.21.15.155:48200) 2024-03-25T18:26:59.681 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.680+0000 7fac49ec0640 1 -- 172.21.15.155:0/2742124594 learned_addr learned my addr 172.21.15.155:0/2742124594 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:26:59.689 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.688+0000 7fac496bf640 1 -- 172.21.15.155:0/2742124594 >> v1:172.21.15.167:6789/0 conn(0x7fac4406ab80 legacy=0x7fac440ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-25T18:26:59.689 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.688+0000 7fac496bf640 1 -- 172.21.15.155:0/2742124594 >> v1:172.21.15.167:6789/0 conn(0x7fac4406ab80 legacy=0x7fac440ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-25T18:26:59.689 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.688+0000 7fac496bf640 1 --1- 172.21.15.155:0/2742124594 >> v1:172.21.15.167:6789/0 conn(0x7fac4406ab80 0x7fac440ad880 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-03-25T18:26:59.694 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.693+0000 7fac3bfff640 1 -- 172.21.15.155:0/2742124594 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2019803949 0 0) 0x7fac440af520 con 0x7fac440b00b0 2024-03-25T18:26:59.695 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.693+0000 7fac3bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:26:59.695 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.693+0000 7fac3bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:59.695 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.693+0000 7fac3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:59.695 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.693+0000 7fac3bfff640 10 monclient(hunting): my global_id is 4325 2024-03-25T18:26:59.695 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.693+0000 7fac3bfff640 1 -- 172.21.15.155:0/2742124594 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac30003520 con 0x7fac440b00b0 2024-03-25T18:26:59.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.713+0000 7fac3bfff640 1 -- 172.21.15.155:0/2742124594 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2535148371 0 0) 0x7fac30003520 con 0x7fac440b00b0 2024-03-25T18:26:59.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.713+0000 7fac3bfff640 1 -- 172.21.15.155:0/2742124594 >> v1:172.21.15.167:6789/0 conn(0x7fac4406ab80 legacy=0x7fac440ad880 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:26:59.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.713+0000 7fac3bfff640 1 -- 172.21.15.155:0/2742124594 >> v1:172.21.15.155:6789/0 conn(0x7fac4406af60 legacy=0x7fac4413f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:59.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.713+0000 7fac3bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:26:59.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.713+0000 7fac3bfff640 1 monclient: found mon.noname-b 2024-03-25T18:26:59.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.713+0000 7fac3bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:59.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.713+0000 7fac3bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-25T18:26:59.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.714+0000 7fac3bfff640 1 -- 172.21.15.155:0/2742124594 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac44072890 con 0x7fac440b00b0 2024-03-25T18:26:59.716 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.714+0000 7fac3bfff640 10 monclient: _finish_auth 0 2024-03-25T18:26:59.716 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.714+0000 7fac3bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:26:59.716 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.714+0000 7fac3bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:59.716 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.714+0000 7fac3bfff640 1 -- 172.21.15.155:0/2742124594 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fac3c001a10 con 0x7fac440b00b0 2024-03-25T18:26:59.716 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.714+0000 7fac4b923640 5 monclient: authenticate success, global_id 4325 2024-03-25T18:26:59.716 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.714+0000 7fac3bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:59.716 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.714+0000 7fac3bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:59.716 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.714+0000 7fac3bfff640 10 monclient: dump: 2024-03-25T18:26:59.716 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:26:59.716 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:26:59.716 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:59.716 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:59.716 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:59.716 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:26:59.716 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:26:59.717 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:26:59.717 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:26:59.717 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:26:59.717 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.714+0000 7fac4b923640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:59.735 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.734+0000 7fac3bfff640 1 -- 172.21.15.155:0/2742124594 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fac3c001be0 con 0x7fac440b00b0 2024-03-25T18:26:59.735 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.734+0000 7fac3bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:59.735 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.734+0000 7fac3bfff640 1 -- 172.21.15.155:0/2742124594 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fac3c002030 con 0x7fac440b00b0 2024-03-25T18:26:59.735 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.734+0000 7fac4b923640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:59.735 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.734+0000 7fac4b923640 4 set_mon_vals no callback set 2024-03-25T18:26:59.735 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.734+0000 7fac3bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:59.735 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.734+0000 7fac3bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:59.735 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.734+0000 7fac3bfff640 10 monclient: dump: 2024-03-25T18:26:59.735 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:26:59.736 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:26:59.736 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:59.736 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:59.736 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:59.736 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:26:59.736 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:26:59.736 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:26:59.736 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:26:59.736 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:26:59.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.735+0000 7fac4b923640 10 monclient: shutdown 2024-03-25T18:26:59.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.735+0000 7fac4b923640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:59.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.735+0000 7fac4b923640 1 -- 172.21.15.155:0/2742124594 >> v1:172.21.15.155:6790/0 conn(0x7fac440b00b0 legacy=0x7fac4414dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:59.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.735+0000 7fac4b923640 1 -- 172.21.15.155:0/2742124594 shutdown_connections 2024-03-25T18:26:59.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.735+0000 7fac4b923640 1 -- 172.21.15.155:0/2742124594 >> 172.21.15.155:0/2742124594 conn(0x7fac440b4a80 msgr2=0x7fac440b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:59.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.735+0000 7fac4b923640 1 -- 172.21.15.155:0/2742124594 shutdown_connections 2024-03-25T18:26:59.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.735+0000 7fac4b923640 1 -- 172.21.15.155:0/2742124594 wait complete. 2024-03-25T18:26:59.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.736+0000 7fac4b923640 10 monclient: build_initial_monmap 2024-03-25T18:26:59.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.736+0000 7fac4b923640 10 monclient: monmap: 2024-03-25T18:26:59.739 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:26:59.739 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:59.739 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:26:59.739 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:26:59.739 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:59.739 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:26:59.739 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:26:59.740 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:26:59.740 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:26:59.740 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:26:59.740 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.736+0000 7fac4b923640 1 librados: starting msgr at 2024-03-25T18:26:59.740 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.736+0000 7fac4b923640 1 librados: starting objecter 2024-03-25T18:26:59.740 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.736+0000 7fac4b923640 1 Processor -- start 2024-03-25T18:26:59.740 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.736+0000 7fac4b923640 1 -- start start 2024-03-25T18:26:59.740 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.736+0000 7fac4b923640 1 librados: setting wanted keys 2024-03-25T18:26:59.740 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.736+0000 7fac4b923640 1 librados: calling monclient init 2024-03-25T18:26:59.740 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.736+0000 7fac4b923640 10 monclient: init 2024-03-25T18:26:59.740 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.737+0000 7fac4b923640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:59.740 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.737+0000 7fac4b923640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:26:59.740 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.737+0000 7fac4b923640 10 monclient(hunting): picked mon.noname-a con 0x7fac4406ab80 addr v1:172.21.15.155:6789/0 2024-03-25T18:26:59.740 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.737+0000 7fac4b923640 10 monclient(hunting): picked mon.noname-c con 0x7fac440b00b0 addr v1:172.21.15.155:6790/0 2024-03-25T18:26:59.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.737+0000 7fac4b923640 10 monclient(hunting): picked mon.noname-b con 0x7fac441401c0 addr v1:172.21.15.167:6789/0 2024-03-25T18:26:59.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.737+0000 7fac4b923640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac441d7cf0 con 0x7fac4406ab80 2024-03-25T18:26:59.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.737+0000 7fac4b923640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac44143540 con 0x7fac441401c0 2024-03-25T18:26:59.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.737+0000 7fac4b923640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac44146700 con 0x7fac440b00b0 2024-03-25T18:26:59.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.737+0000 7fac4b923640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:59.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.737+0000 7fac4b923640 10 monclient(hunting): authenticate will time out at 7666.047273s 2024-03-25T18:26:59.749 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.748+0000 7fac496bf640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fac4406ab80 0x7fac44142e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:48898/0 (socket says 172.21.15.155:48898) 2024-03-25T18:26:59.749 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.748+0000 7fac496bf640 1 -- 172.21.15.155:0/2799142079 learned_addr learned my addr 172.21.15.155:0/2799142079 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:26:59.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.751+0000 7fac39ffb640 10 client.?.objecter ms_handle_connect 0x7fac4406ab80 2024-03-25T18:26:59.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.751+0000 7fac39ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:59.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.756+0000 7fac39ffb640 10 client.?.objecter ms_handle_connect 0x7fac440b00b0 2024-03-25T18:26:59.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.756+0000 7fac39ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:59.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.761+0000 7fac39ffb640 1 -- 172.21.15.155:0/2799142079 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1660372898 0 0) 0x7fac441d7cf0 con 0x7fac4406ab80 2024-03-25T18:26:59.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.761+0000 7fac39ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:26:59.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.761+0000 7fac39ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:59.763 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.761+0000 7fac39ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:59.763 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.761+0000 7fac39ffb640 10 monclient(hunting): my global_id is 4323 2024-03-25T18:26:59.763 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.761+0000 7fac39ffb640 1 -- 172.21.15.155:0/2799142079 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac28004eb0 con 0x7fac4406ab80 2024-03-25T18:26:59.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.768+0000 7fac39ffb640 1 -- 172.21.15.155:0/2799142079 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2263334050 0 0) 0x7fac44146700 con 0x7fac440b00b0 2024-03-25T18:26:59.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.768+0000 7fac39ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:26:59.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.768+0000 7fac39ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:59.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.768+0000 7fac39ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:59.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.768+0000 7fac39ffb640 10 monclient(hunting): my global_id is 4328 2024-03-25T18:26:59.770 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.768+0000 7fac39ffb640 1 -- 172.21.15.155:0/2799142079 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac441d7cf0 con 0x7fac440b00b0 2024-03-25T18:26:59.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.775+0000 7fac39ffb640 10 client.?.objecter ms_handle_connect 0x7fac441401c0 2024-03-25T18:26:59.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.775+0000 7fac39ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:59.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.783+0000 7fac39ffb640 1 -- 172.21.15.155:0/2799142079 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2042513401 0 0) 0x7fac44143540 con 0x7fac441401c0 2024-03-25T18:26:59.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.783+0000 7fac39ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:26:59.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.783+0000 7fac39ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:59.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.783+0000 7fac39ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:59.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.783+0000 7fac39ffb640 10 monclient(hunting): my global_id is 4318 2024-03-25T18:26:59.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.783+0000 7fac39ffb640 1 -- 172.21.15.155:0/2799142079 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac44146700 con 0x7fac441401c0 2024-03-25T18:26:59.790 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.788+0000 7fac39ffb640 1 -- 172.21.15.155:0/2799142079 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2714013132 0 0) 0x7fac28004eb0 con 0x7fac4406ab80 2024-03-25T18:26:59.790 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.788+0000 7fac39ffb640 1 -- 172.21.15.155:0/2799142079 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac44143540 con 0x7fac4406ab80 2024-03-25T18:26:59.790 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.788+0000 7fac39ffb640 1 -- 172.21.15.155:0/2799142079 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fac400040a0 con 0x7fac4406ab80 2024-03-25T18:26:59.790 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.788+0000 7fac39ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:59.790 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.788+0000 7fac39ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:26:59.790 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.788+0000 7fac39ffb640 10 monclient(hunting): dump: 2024-03-25T18:26:59.791 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:26:59.791 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:26:59.791 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:59.791 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:59.791 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:59.791 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:26:59.791 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:26:59.791 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:26:59.791 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:26:59.791 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:26:59.791 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.788+0000 7fac39ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:59.794 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.793+0000 7fac39ffb640 1 -- 172.21.15.155:0/2799142079 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3586637217 0 0) 0x7fac441d7cf0 con 0x7fac440b00b0 2024-03-25T18:26:59.794 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.793+0000 7fac39ffb640 1 -- 172.21.15.155:0/2799142079 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac28004eb0 con 0x7fac440b00b0 2024-03-25T18:26:59.794 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.793+0000 7fac39ffb640 1 -- 172.21.15.155:0/2799142079 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fac34001930 con 0x7fac440b00b0 2024-03-25T18:26:59.794 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.793+0000 7fac39ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:59.794 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.793+0000 7fac39ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:59.794 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.793+0000 7fac39ffb640 10 monclient(hunting): dump: 2024-03-25T18:26:59.794 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:26:59.794 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:26:59.794 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:59.794 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:59.794 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:59.794 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:26:59.795 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:26:59.795 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:26:59.795 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:26:59.795 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:26:59.797 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.796+0000 7fac39ffb640 1 -- 172.21.15.155:0/2799142079 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2924645548 0 0) 0x7fac44146700 con 0x7fac441401c0 2024-03-25T18:26:59.797 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.796+0000 7fac39ffb640 1 -- 172.21.15.155:0/2799142079 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac441d7cf0 con 0x7fac441401c0 2024-03-25T18:26:59.797 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.796+0000 7fac39ffb640 1 -- 172.21.15.155:0/2799142079 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fac3c001d50 con 0x7fac441401c0 2024-03-25T18:26:59.797 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.796+0000 7fac39ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:59.797 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.796+0000 7fac39ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:59.797 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.796+0000 7fac39ffb640 10 monclient(hunting): dump: 2024-03-25T18:26:59.797 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:26:59.797 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:26:59.797 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:59.797 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:26:59.798 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:59.798 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:26:59.798 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:26:59.798 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:26:59.798 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:26:59.798 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:26:59.804 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.802+0000 7fac39ffb640 1 -- 172.21.15.155:0/2799142079 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1874138566 0 0) 0x7fac28004eb0 con 0x7fac440b00b0 2024-03-25T18:26:59.804 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.802+0000 7fac39ffb640 1 -- 172.21.15.155:0/2799142079 >> v1:172.21.15.167:6789/0 conn(0x7fac441401c0 legacy=0x7fac441e2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:59.804 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.802+0000 7fac39ffb640 1 -- 172.21.15.155:0/2799142079 >> v1:172.21.15.155:6789/0 conn(0x7fac4406ab80 legacy=0x7fac44142e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:59.804 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.802+0000 7fac39ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:26:59.804 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.802+0000 7fac39ffb640 1 monclient: found mon.c 2024-03-25T18:26:59.804 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.802+0000 7fac39ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:59.804 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.802+0000 7fac39ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:26:59.804 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.802+0000 7fac39ffb640 1 -- 172.21.15.155:0/2799142079 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac441d8890 con 0x7fac440b00b0 2024-03-25T18:26:59.804 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.802+0000 7fac39ffb640 10 monclient: _finish_auth 0 2024-03-25T18:26:59.804 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.802+0000 7fac39ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:26:59.804 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.802+0000 7fac39ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:59.804 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.802+0000 7fac4b923640 5 monclient: authenticate success, global_id 4328 2024-03-25T18:26:59.804 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.802+0000 7fac4b923640 10 monclient: _renew_subs 2024-03-25T18:26:59.805 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.802+0000 7fac4b923640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:26:59.805 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.802+0000 7fac4b923640 1 -- 172.21.15.155:0/2799142079 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac44137e20 con 0x7fac440b00b0 2024-03-25T18:26:59.805 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.803+0000 7fac4b923640 10 client.4328.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:59.805 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.803+0000 7fac4b923640 10 monclient: _renew_subs 2024-03-25T18:26:59.805 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.803+0000 7fac4b923640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:26:59.805 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.803+0000 7fac4b923640 1 -- 172.21.15.155:0/2799142079 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fac441e3c20 con 0x7fac440b00b0 2024-03-25T18:26:59.805 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.803+0000 7fac4b923640 1 librados: init done 2024-03-25T18:26:59.805 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.804+0000 7fac4b923640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:59.806 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.804+0000 7fac4b923640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:59.806 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.804+0000 7fac4b923640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:26:59.806 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:26:59.804+0000 7fac4b923640 1 -- 172.21.15.155:0/2799142079 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac441d7ac0 con 0x7fac440b00b0 2024-03-25T18:27:00.097 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:00.096+0000 7fac39ffb640 1 -- 172.21.15.155:0/2799142079 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fac340035f0 con 0x7fac440b00b0 2024-03-25T18:27:00.097 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:00.096+0000 7fac39ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:00.097 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:00.096+0000 7fac39ffb640 1 -- 172.21.15.155:0/2799142079 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fac340037a0 con 0x7fac440b00b0 2024-03-25T18:27:00.097 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:00.096+0000 7fac39ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:00.097 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:00.096+0000 7fac39ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:00.098 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:00.096+0000 7fac39ffb640 10 monclient: dump: 2024-03-25T18:27:00.098 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:27:00.098 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:27:00.098 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:00.098 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:00.098 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:00.098 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:27:00.098 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:27:00.098 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:27:00.098 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:27:00.098 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:27:00.098 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:00.096+0000 7fac3b7fe640 4 set_mon_vals no callback set 2024-03-25T18:27:00.099 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:00.097+0000 7fac39ffb640 1 -- 172.21.15.155:0/2799142079 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7fac340039c0 con 0x7fac440b00b0 2024-03-25T18:27:00.099 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:00.098+0000 7fac496bf640 10 monclient: get_auth_request con 0x7fac2806e110 auth_method 0 2024-03-25T18:27:00.099 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:00.098+0000 7fac39ffb640 10 client.4328.objecter ms_handle_connect 0x7fac2806e110 2024-03-25T18:27:00.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:00.102+0000 7fac39ffb640 1 -- 172.21.15.155:0/2799142079 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7fac34053470 con 0x7fac440b00b0 2024-03-25T18:27:00.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:00.102+0000 7fac39ffb640 10 client.4328.objecter ms_dispatch 0x7fac44000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:00.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:00.102+0000 7fac39ffb640 3 client.4328.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:00.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:00.102+0000 7fac39ffb640 3 client.4328.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:00.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:00.102+0000 7fac39ffb640 20 client.4328.objecter dump_active .. 0 homeless 2024-03-25T18:27:00.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:00.102+0000 7fac39ffb640 1 -- 172.21.15.155:0/2799142079 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fac3404ef80 con 0x7fac440b00b0 2024-03-25T18:27:00.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:00.102+0000 7fac39ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:00.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:00.102+0000 7fac39ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:00.335 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:00.334+0000 7fac4b923640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:00.336 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:00.334+0000 7fac4b923640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:00.336 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:00.334+0000 7fac4b923640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:27:00.336 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:00.334+0000 7fac4b923640 1 -- 172.21.15.155:0/2799142079 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fac440627b0 con 0x7fac440b00b0 2024-03-25T18:27:00.366 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:00.365+0000 7fac39ffb640 1 -- 172.21.15.155:0/2799142079 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7fac34050eb0 con 0x7fac440b00b0 2024-03-25T18:27:00.367 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:00.365+0000 7fac39ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:00.367 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:00.365+0000 7fac39ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:00.367 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:27:00.367 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:00.369 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:00.368+0000 7fac4b923640 10 librados: watch_flush enter 2024-03-25T18:27:00.369 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:00.368+0000 7fac4b923640 10 librados: watch_flush exit 2024-03-25T18:27:00.369 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:00.368+0000 7fac4b923640 20 client.4328.objecter shutdown clearing up homeless session... 2024-03-25T18:27:00.369 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:00.368+0000 7fac4b923640 10 client.4328.objecter successfully canceled tick 2024-03-25T18:27:00.369 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:00.368+0000 7fac4b923640 1 -- 172.21.15.155:0/2799142079 >> v1:172.21.15.167:6800/2919070503 conn(0x7fac2806e110 legacy=0x7fac280705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:00.369 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:00.368+0000 7fac4b923640 10 monclient: shutdown 2024-03-25T18:27:00.369 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:00.368+0000 7fac4b923640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:00.369 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:00.368+0000 7fac4b923640 1 -- 172.21.15.155:0/2799142079 >> v1:172.21.15.155:6790/0 conn(0x7fac440b00b0 legacy=0x7fac441d8180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:00.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:00.368+0000 7fac4b923640 1 -- 172.21.15.155:0/2799142079 shutdown_connections 2024-03-25T18:27:00.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:00.368+0000 7fac4b923640 1 -- 172.21.15.155:0/2799142079 >> 172.21.15.155:0/2799142079 conn(0x7fac440b4a80 msgr2=0x7fac440ad880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:00.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:00.368+0000 7fac4b923640 1 -- 172.21.15.155:0/2799142079 shutdown_connections 2024-03-25T18:27:00.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:00.368+0000 7fac4b923640 1 -- 172.21.15.155:0/2799142079 wait complete. 2024-03-25T18:27:00.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:00.368+0000 7fac4b923640 1 librados: shutdown 2024-03-25T18:27:00.381 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:00.726 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:02:34.330 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:00.732 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:02:34.311 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:00.739 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:02:34.343 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:00.768 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:02:34.373 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:00.786 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:02:34.364 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:00.790 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:02:34.395 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:00.794 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:02:34.399 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:00.824 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:02:34.429 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:00.827 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:02:34.406 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:07.188 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:07.295 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.294+0000 7f36292f1640 10 monclient: get_monmap_and_config 2024-03-25T18:27:07.296 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.294+0000 7f36292f1640 10 monclient: build_initial_monmap 2024-03-25T18:27:07.296 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.294+0000 7f36292f1640 10 monclient: monmap: 2024-03-25T18:27:07.296 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:27:07.296 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:27:07.296 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:27:07.295697+0000 2024-03-25T18:27:07.296 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:27:07.295697+0000 2024-03-25T18:27:07.296 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:07.296 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:27:07.296 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:27:07.296 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:27:07.296 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:27:07.296 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:27:07.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.295+0000 7f36292f1640 1 Processor -- start 2024-03-25T18:27:07.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.295+0000 7f36292f1640 1 -- start start 2024-03-25T18:27:07.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.295+0000 7f36292f1640 10 monclient: init 2024-03-25T18:27:07.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.296+0000 7f36292f1640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:07.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.296+0000 7f36292f1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:27:07.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.296+0000 7f36292f1640 10 monclient(hunting): picked mon.noname-b con 0x7f36240a98f0 addr v1:172.21.15.155:6790/0 2024-03-25T18:27:07.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.296+0000 7f36292f1640 10 monclient(hunting): picked mon.noname-a con 0x7f362413f710 addr v1:172.21.15.155:6789/0 2024-03-25T18:27:07.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.296+0000 7f36292f1640 10 monclient(hunting): picked mon.noname-c con 0x7f36241430c0 addr v1:172.21.15.167:6789/0 2024-03-25T18:27:07.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.296+0000 7f36292f1640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3624142520 con 0x7f362413f710 2024-03-25T18:27:07.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.296+0000 7f36292f1640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36240b28a0 con 0x7f36241430c0 2024-03-25T18:27:07.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.296+0000 7f36292f1640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36240b2a30 con 0x7f36240a98f0 2024-03-25T18:27:07.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.296+0000 7f36292f1640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:07.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.296+0000 7f36292f1640 10 monclient(hunting): authenticate will time out at 7673.605983s 2024-03-25T18:27:07.299 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.298+0000 7f36237fe640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f36241430c0 0x7f362414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:37964/0 (socket says 172.21.15.155:37964) 2024-03-25T18:27:07.299 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.298+0000 7f36237fe640 1 -- 172.21.15.155:0/1448079859 learned_addr learned my addr 172.21.15.155:0/1448079859 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:27:07.313 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.311+0000 7f3621ffb640 1 -- 172.21.15.155:0/1448079859 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2156203790 0 0) 0x7f36240b28a0 con 0x7f36241430c0 2024-03-25T18:27:07.313 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.311+0000 7f3621ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:27:07.313 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.311+0000 7f3621ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:07.313 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.311+0000 7f3621ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:07.313 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.311+0000 7f3621ffb640 10 monclient(hunting): my global_id is 4321 2024-03-25T18:27:07.313 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.311+0000 7f3621ffb640 1 -- 172.21.15.155:0/1448079859 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f360c003520 con 0x7f36241430c0 2024-03-25T18:27:07.317 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.316+0000 7f3621ffb640 1 -- 172.21.15.155:0/1448079859 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 368916598 0 0) 0x7f36240b2a30 con 0x7f36240a98f0 2024-03-25T18:27:07.317 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.316+0000 7f3621ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:27:07.317 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.316+0000 7f3621ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:07.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.316+0000 7f3621ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:07.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.316+0000 7f3621ffb640 10 monclient(hunting): my global_id is 4331 2024-03-25T18:27:07.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.316+0000 7f3621ffb640 1 -- 172.21.15.155:0/1448079859 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36240b28a0 con 0x7f36240a98f0 2024-03-25T18:27:07.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.321+0000 7f3621ffb640 1 -- 172.21.15.155:0/1448079859 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1268326767 0 0) 0x7f36240b28a0 con 0x7f36240a98f0 2024-03-25T18:27:07.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.321+0000 7f3621ffb640 1 -- 172.21.15.155:0/1448079859 >> v1:172.21.15.167:6789/0 conn(0x7f36241430c0 legacy=0x7f362414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:07.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.321+0000 7f3621ffb640 1 -- 172.21.15.155:0/1448079859 >> v1:172.21.15.155:6789/0 conn(0x7f362413f710 legacy=0x7f36241429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:07.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.321+0000 7f3621ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:27:07.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.321+0000 7f3621ffb640 1 monclient: found mon.noname-b 2024-03-25T18:27:07.324 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.321+0000 7f3621ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:07.324 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.321+0000 7f3621ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-25T18:27:07.324 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.321+0000 7f3621ffb640 1 -- 172.21.15.155:0/1448079859 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36240b44c0 con 0x7f36240a98f0 2024-03-25T18:27:07.324 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.321+0000 7f3621ffb640 10 monclient: _finish_auth 0 2024-03-25T18:27:07.324 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.321+0000 7f3621ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:27:07.324 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.321+0000 7f3621ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:07.324 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.321+0000 7f3621ffb640 1 -- 172.21.15.155:0/1448079859 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f3614001670 con 0x7f36240a98f0 2024-03-25T18:27:07.324 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.321+0000 7f3621ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:07.324 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.321+0000 7f3621ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:27:07.324 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.321+0000 7f3621ffb640 10 monclient: dump: 2024-03-25T18:27:07.324 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:27:07.324 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:27:07.324 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:07.324 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:07.325 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:07.325 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:27:07.325 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:27:07.325 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:27:07.325 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:27:07.325 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:27:07.325 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.321+0000 7f36292f1640 5 monclient: authenticate success, global_id 4331 2024-03-25T18:27:07.325 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.321+0000 7f36292f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:07.338 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.337+0000 7f3621ffb640 1 -- 172.21.15.155:0/1448079859 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36140017d0 con 0x7f36240a98f0 2024-03-25T18:27:07.338 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.337+0000 7f3621ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:07.338 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.337+0000 7f3621ffb640 1 -- 172.21.15.155:0/1448079859 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f3614001c20 con 0x7f36240a98f0 2024-03-25T18:27:07.338 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.337+0000 7f36292f1640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:07.338 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.337+0000 7f36292f1640 4 set_mon_vals no callback set 2024-03-25T18:27:07.338 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.337+0000 7f3621ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:07.339 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.337+0000 7f3621ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:07.339 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.337+0000 7f3621ffb640 10 monclient: dump: 2024-03-25T18:27:07.339 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:27:07.339 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:27:07.339 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:07.339 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:07.339 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:07.339 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:27:07.339 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:27:07.339 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:27:07.339 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:27:07.339 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:27:07.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.338+0000 7f36292f1640 10 monclient: shutdown 2024-03-25T18:27:07.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.338+0000 7f36292f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:07.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.338+0000 7f36292f1640 1 -- 172.21.15.155:0/1448079859 >> v1:172.21.15.155:6790/0 conn(0x7f36240a98f0 legacy=0x7f36240a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:07.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.338+0000 7f36292f1640 1 -- 172.21.15.155:0/1448079859 shutdown_connections 2024-03-25T18:27:07.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.338+0000 7f36292f1640 1 -- 172.21.15.155:0/1448079859 >> 172.21.15.155:0/1448079859 conn(0x7f36240b7ea0 msgr2=0x7f36240b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:07.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.338+0000 7f36292f1640 1 -- 172.21.15.155:0/1448079859 shutdown_connections 2024-03-25T18:27:07.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.338+0000 7f36292f1640 1 -- 172.21.15.155:0/1448079859 wait complete. 2024-03-25T18:27:07.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.339+0000 7f36292f1640 10 monclient: build_initial_monmap 2024-03-25T18:27:07.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.339+0000 7f36292f1640 10 monclient: monmap: 2024-03-25T18:27:07.342 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:27:07.342 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:07.342 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:27:07.342 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:27:07.342 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:07.342 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:27:07.343 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:27:07.343 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:27:07.343 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:27:07.343 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:27:07.343 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.339+0000 7f36292f1640 1 librados: starting msgr at 2024-03-25T18:27:07.343 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.339+0000 7f36292f1640 1 librados: starting objecter 2024-03-25T18:27:07.343 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.340+0000 7f36292f1640 1 Processor -- start 2024-03-25T18:27:07.343 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.340+0000 7f36292f1640 1 -- start start 2024-03-25T18:27:07.343 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.340+0000 7f36292f1640 1 librados: setting wanted keys 2024-03-25T18:27:07.343 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.340+0000 7f36292f1640 1 librados: calling monclient init 2024-03-25T18:27:07.343 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.340+0000 7f36292f1640 10 monclient: init 2024-03-25T18:27:07.343 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.340+0000 7f36292f1640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:07.343 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.340+0000 7f36292f1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:27:07.343 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.340+0000 7f36292f1640 10 monclient(hunting): picked mon.noname-a con 0x7f36240a98f0 addr v1:172.21.15.155:6789/0 2024-03-25T18:27:07.344 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.340+0000 7f36292f1640 10 monclient(hunting): picked mon.noname-b con 0x7f362413f710 addr v1:172.21.15.167:6789/0 2024-03-25T18:27:07.344 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.340+0000 7f36292f1640 10 monclient(hunting): picked mon.noname-c con 0x7f36241430c0 addr v1:172.21.15.155:6790/0 2024-03-25T18:27:07.344 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.340+0000 7f36292f1640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36241390e0 con 0x7f36240a98f0 2024-03-25T18:27:07.344 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.340+0000 7f36292f1640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36241349e0 con 0x7f362413f710 2024-03-25T18:27:07.344 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.340+0000 7f36292f1640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3624139350 con 0x7f36241430c0 2024-03-25T18:27:07.344 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.340+0000 7f36292f1640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:07.344 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.340+0000 7f36292f1640 10 monclient(hunting): authenticate will time out at 7673.650579s 2024-03-25T18:27:07.344 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.343+0000 7f36227fc640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f362413f710 0x7f3624133bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:37994/0 (socket says 172.21.15.155:37994) 2024-03-25T18:27:07.344 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.344+0000 7f36227fc640 1 -- 172.21.15.155:0/2340227723 learned_addr learned my addr 172.21.15.155:0/2340227723 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:27:07.345 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.345+0000 7f360b7fe640 10 client.?.objecter ms_handle_connect 0x7f362413f710 2024-03-25T18:27:07.345 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.345+0000 7f360b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:07.354 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.352+0000 7f360b7fe640 1 -- 172.21.15.155:0/2340227723 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 325220752 0 0) 0x7f36241349e0 con 0x7f362413f710 2024-03-25T18:27:07.354 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.352+0000 7f360b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:27:07.354 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.352+0000 7f360b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:07.354 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.352+0000 7f360b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:07.354 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.352+0000 7f360b7fe640 10 monclient(hunting): my global_id is 4324 2024-03-25T18:27:07.354 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.353+0000 7f360b7fe640 1 -- 172.21.15.155:0/2340227723 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3610004eb0 con 0x7f362413f710 2024-03-25T18:27:07.355 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.354+0000 7f360b7fe640 10 client.?.objecter ms_handle_connect 0x7f36240a98f0 2024-03-25T18:27:07.355 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.354+0000 7f360b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:07.367 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.366+0000 7f360b7fe640 1 -- 172.21.15.155:0/2340227723 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 338032827 0 0) 0x7f3610004eb0 con 0x7f362413f710 2024-03-25T18:27:07.367 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.366+0000 7f360b7fe640 1 -- 172.21.15.155:0/2340227723 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36241349e0 con 0x7f362413f710 2024-03-25T18:27:07.367 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.366+0000 7f360b7fe640 1 -- 172.21.15.155:0/2340227723 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f3618001a30 con 0x7f362413f710 2024-03-25T18:27:07.367 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.366+0000 7f360b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:07.367 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.366+0000 7f360b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:27:07.367 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.366+0000 7f360b7fe640 10 monclient(hunting): dump: 2024-03-25T18:27:07.367 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:27:07.367 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:27:07.367 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:07.368 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:07.368 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:07.368 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:27:07.368 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:27:07.368 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:27:07.368 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:27:07.368 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:27:07.368 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.366+0000 7f360b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:07.368 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.367+0000 7f360b7fe640 10 client.?.objecter ms_handle_connect 0x7f36241430c0 2024-03-25T18:27:07.368 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.367+0000 7f360b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:07.374 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.372+0000 7f360b7fe640 1 -- 172.21.15.155:0/2340227723 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4289395467 0 0) 0x7f36241390e0 con 0x7f36240a98f0 2024-03-25T18:27:07.374 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.372+0000 7f360b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:27:07.374 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.372+0000 7f360b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:07.374 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.372+0000 7f360b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:07.375 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.372+0000 7f360b7fe640 10 monclient(hunting): my global_id is 4329 2024-03-25T18:27:07.375 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.372+0000 7f360b7fe640 1 -- 172.21.15.155:0/2340227723 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3610004eb0 con 0x7f36240a98f0 2024-03-25T18:27:07.386 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.385+0000 7f360b7fe640 1 -- 172.21.15.155:0/2340227723 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3538993497 0 0) 0x7f3624139350 con 0x7f36241430c0 2024-03-25T18:27:07.387 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.385+0000 7f360b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:27:07.387 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.385+0000 7f360b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:07.387 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.385+0000 7f360b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:07.387 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.385+0000 7f360b7fe640 10 monclient(hunting): my global_id is 4334 2024-03-25T18:27:07.387 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.385+0000 7f360b7fe640 1 -- 172.21.15.155:0/2340227723 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36241390e0 con 0x7f36241430c0 2024-03-25T18:27:07.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.385+0000 7f360b7fe640 1 -- 172.21.15.155:0/2340227723 <== mon.1 v1:172.21.15.167:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3840707418 0 0) 0x7f36241349e0 con 0x7f362413f710 2024-03-25T18:27:07.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.385+0000 7f360b7fe640 1 -- 172.21.15.155:0/2340227723 >> v1:172.21.15.155:6790/0 conn(0x7f36241430c0 legacy=0x7f36241342d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:07.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.385+0000 7f360b7fe640 1 -- 172.21.15.155:0/2340227723 >> v1:172.21.15.155:6789/0 conn(0x7f36240a98f0 legacy=0x7f36241d72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:07.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.385+0000 7f360b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:27:07.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.385+0000 7f360b7fe640 1 monclient: found mon.b 2024-03-25T18:27:07.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.385+0000 7f360b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:07.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.385+0000 7f360b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:27:07.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.385+0000 7f360b7fe640 1 -- 172.21.15.155:0/2340227723 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3624139630 con 0x7f362413f710 2024-03-25T18:27:07.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.385+0000 7f360b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:27:07.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.385+0000 7f360b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:27:07.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.385+0000 7f360b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:07.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.385+0000 7f36292f1640 5 monclient: authenticate success, global_id 4324 2024-03-25T18:27:07.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.385+0000 7f36292f1640 10 monclient: _renew_subs 2024-03-25T18:27:07.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.385+0000 7f36292f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:27:07.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.385+0000 7f36292f1640 1 -- 172.21.15.155:0/2340227723 --> v1:172.21.15.167:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3624139910 con 0x7f362413f710 2024-03-25T18:27:07.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.386+0000 7f36292f1640 10 client.4324.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:07.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.386+0000 7f36292f1640 10 monclient: _renew_subs 2024-03-25T18:27:07.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.386+0000 7f36292f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:27:07.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.386+0000 7f36292f1640 1 -- 172.21.15.155:0/2340227723 --> v1:172.21.15.167:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3624137800 con 0x7f362413f710 2024-03-25T18:27:07.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.386+0000 7f36292f1640 1 librados: init done 2024-03-25T18:27:07.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.387+0000 7f36292f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:07.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.387+0000 7f36292f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:07.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.387+0000 7f36292f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:27:07.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.387+0000 7f36292f1640 1 -- 172.21.15.155:0/2340227723 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3624138410 con 0x7f362413f710 2024-03-25T18:27:07.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.663+0000 7f360b7fe640 1 -- 172.21.15.155:0/2340227723 <== mon.1 v1:172.21.15.167:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3618001cf0 con 0x7f362413f710 2024-03-25T18:27:07.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.663+0000 7f360b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:07.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.663+0000 7f360b7fe640 1 -- 172.21.15.155:0/2340227723 <== mon.1 v1:172.21.15.167:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f3618003a60 con 0x7f362413f710 2024-03-25T18:27:07.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.663+0000 7f360b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:07.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.663+0000 7f360b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:07.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.663+0000 7f36217fa640 4 set_mon_vals no callback set 2024-03-25T18:27:07.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.663+0000 7f360b7fe640 10 monclient: dump: 2024-03-25T18:27:07.666 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:27:07.666 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:27:07.666 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:07.666 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:07.666 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:07.666 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:27:07.666 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:27:07.666 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:27:07.666 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:27:07.666 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:27:07.667 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.664+0000 7f360b7fe640 1 -- 172.21.15.155:0/2340227723 <== mon.1 v1:172.21.15.167:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7f3618003c80 con 0x7f362413f710 2024-03-25T18:27:07.667 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.664+0000 7f3622ffd640 10 monclient: get_auth_request con 0x7f361006de10 auth_method 0 2024-03-25T18:27:07.667 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.665+0000 7f360b7fe640 10 client.4324.objecter ms_handle_connect 0x7f361006de10 2024-03-25T18:27:07.670 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.668+0000 7f360b7fe640 1 -- 172.21.15.155:0/2340227723 <== mon.1 v1:172.21.15.167:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7f36180548f0 con 0x7f362413f710 2024-03-25T18:27:07.670 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.668+0000 7f360b7fe640 10 client.4324.objecter ms_dispatch 0x7f3624000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:07.670 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.668+0000 7f360b7fe640 3 client.4324.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:07.670 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.668+0000 7f360b7fe640 3 client.4324.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:07.670 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.668+0000 7f360b7fe640 20 client.4324.objecter dump_active .. 0 homeless 2024-03-25T18:27:07.674 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.672+0000 7f360b7fe640 1 -- 172.21.15.155:0/2340227723 <== mon.1 v1:172.21.15.167:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3618050400 con 0x7f362413f710 2024-03-25T18:27:07.674 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.672+0000 7f360b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:07.674 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.672+0000 7f360b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:07.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.924+0000 7f36292f1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:07.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.924+0000 7f36292f1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:07.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.925+0000 7f36292f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:27:07.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.925+0000 7f36292f1640 1 -- 172.21.15.155:0/2340227723 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f36240627b0 con 0x7f362413f710 2024-03-25T18:27:07.951 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.949+0000 7f360b7fe640 1 -- 172.21.15.155:0/2340227723 <== mon.1 v1:172.21.15.167:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7f3618052330 con 0x7f362413f710 2024-03-25T18:27:07.952 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.949+0000 7f360b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:07.952 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.949+0000 7f360b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:07.952 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:27:07.952 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:07.954 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.952+0000 7f36292f1640 10 librados: watch_flush enter 2024-03-25T18:27:07.954 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.952+0000 7f36292f1640 10 librados: watch_flush exit 2024-03-25T18:27:07.954 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.952+0000 7f36292f1640 20 client.4324.objecter shutdown clearing up homeless session... 2024-03-25T18:27:07.954 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.952+0000 7f36292f1640 10 client.4324.objecter successfully canceled tick 2024-03-25T18:27:07.955 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.952+0000 7f36292f1640 1 -- 172.21.15.155:0/2340227723 >> v1:172.21.15.167:6800/2919070503 conn(0x7f361006de10 legacy=0x7f36100702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:07.955 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.952+0000 7f36292f1640 10 monclient: shutdown 2024-03-25T18:27:07.955 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.952+0000 7f36292f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:07.955 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.952+0000 7f36292f1640 1 -- 172.21.15.155:0/2340227723 >> v1:172.21.15.167:6789/0 conn(0x7f362413f710 legacy=0x7f3624133bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:07.955 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.952+0000 7f36292f1640 1 -- 172.21.15.155:0/2340227723 shutdown_connections 2024-03-25T18:27:07.955 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.952+0000 7f36292f1640 1 -- 172.21.15.155:0/2340227723 >> 172.21.15.155:0/2340227723 conn(0x7f36240b7ea0 msgr2=0x7f3624074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:07.955 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.952+0000 7f36292f1640 1 -- 172.21.15.155:0/2340227723 shutdown_connections 2024-03-25T18:27:07.955 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.953+0000 7f36292f1640 1 -- 172.21.15.155:0/2340227723 wait complete. 2024-03-25T18:27:07.955 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:07.953+0000 7f36292f1640 1 librados: shutdown 2024-03-25T18:27:07.968 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:14.775 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:14.891 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.890+0000 7f563a00e640 10 monclient: get_monmap_and_config 2024-03-25T18:27:14.891 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.890+0000 7f563a00e640 10 monclient: build_initial_monmap 2024-03-25T18:27:14.891 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.890+0000 7f563a00e640 10 monclient: monmap: 2024-03-25T18:27:14.891 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:27:14.892 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:27:14.892 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:27:14.891457+0000 2024-03-25T18:27:14.892 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:27:14.891457+0000 2024-03-25T18:27:14.892 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:14.892 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:27:14.892 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:27:14.892 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:27:14.892 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:27:14.892 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:27:14.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.891+0000 7f563a00e640 1 Processor -- start 2024-03-25T18:27:14.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.891+0000 7f563a00e640 1 -- start start 2024-03-25T18:27:14.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.891+0000 7f563a00e640 10 monclient: init 2024-03-25T18:27:14.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.891+0000 7f563a00e640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:14.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.891+0000 7f563a00e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:27:14.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.891+0000 7f563a00e640 10 monclient(hunting): picked mon.noname-a con 0x7f56340a98f0 addr v1:172.21.15.155:6789/0 2024-03-25T18:27:14.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.892+0000 7f563a00e640 10 monclient(hunting): picked mon.noname-c con 0x7f563413f710 addr v1:172.21.15.167:6789/0 2024-03-25T18:27:14.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.892+0000 7f563a00e640 10 monclient(hunting): picked mon.noname-b con 0x7f56341430c0 addr v1:172.21.15.155:6790/0 2024-03-25T18:27:14.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.892+0000 7f563a00e640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5634142520 con 0x7f56340a98f0 2024-03-25T18:27:14.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.892+0000 7f563a00e640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56340b28a0 con 0x7f563413f710 2024-03-25T18:27:14.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.892+0000 7f563a00e640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56340b2a30 con 0x7f56341430c0 2024-03-25T18:27:14.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.892+0000 7f563a00e640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:14.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.892+0000 7f563a00e640 10 monclient(hunting): authenticate will time out at 7681.201898s 2024-03-25T18:27:14.896 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.895+0000 7f56337fe640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f56340a98f0 0x7f56340a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:35286/0 (socket says 172.21.15.155:35286) 2024-03-25T18:27:14.896 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.895+0000 7f56337fe640 1 -- 172.21.15.155:0/2629126383 learned_addr learned my addr 172.21.15.155:0/2629126383 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:27:14.907 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.906+0000 7f56327fc640 1 -- 172.21.15.155:0/2629126383 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2769872432 0 0) 0x7f5634142520 con 0x7f56340a98f0 2024-03-25T18:27:14.907 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.906+0000 7f56327fc640 10 monclient(hunting): _negotiate 2024-03-25T18:27:14.907 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.906+0000 7f56327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:14.907 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.906+0000 7f56327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:14.907 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.906+0000 7f56327fc640 10 monclient(hunting): my global_id is 4332 2024-03-25T18:27:14.907 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.906+0000 7f56327fc640 1 -- 172.21.15.155:0/2629126383 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5620003520 con 0x7f56340a98f0 2024-03-25T18:27:14.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.916+0000 7f56327fc640 1 -- 172.21.15.155:0/2629126383 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3148292055 0 0) 0x7f5620003520 con 0x7f56340a98f0 2024-03-25T18:27:14.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.916+0000 7f56327fc640 1 -- 172.21.15.155:0/2629126383 >> v1:172.21.15.155:6790/0 conn(0x7f56341430c0 legacy=0x7f563414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:14.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.916+0000 7f56327fc640 1 -- 172.21.15.155:0/2629126383 >> v1:172.21.15.167:6789/0 conn(0x7f563413f710 legacy=0x7f56341429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:27:14.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.916+0000 7f56327fc640 10 monclient: _finish_hunting 0 2024-03-25T18:27:14.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.916+0000 7f56327fc640 1 monclient: found mon.noname-a 2024-03-25T18:27:14.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.916+0000 7f56327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:14.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.916+0000 7f56327fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-25T18:27:14.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.916+0000 7f56327fc640 1 -- 172.21.15.155:0/2629126383 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56340b44c0 con 0x7f56340a98f0 2024-03-25T18:27:14.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.916+0000 7f56327fc640 10 monclient: _finish_auth 0 2024-03-25T18:27:14.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.916+0000 7f56327fc640 10 monclient: _check_auth_tickets 2024-03-25T18:27:14.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.916+0000 7f56327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:14.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.916+0000 7f56327fc640 1 -- 172.21.15.155:0/2629126383 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f5624001690 con 0x7f56340a98f0 2024-03-25T18:27:14.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.916+0000 7f563a00e640 5 monclient: authenticate success, global_id 4332 2024-03-25T18:27:14.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.916+0000 7f56327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:14.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.916+0000 7f56327fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:27:14.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.916+0000 7f56327fc640 10 monclient: dump: 2024-03-25T18:27:14.918 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:27:14.919 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:27:14.919 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:14.919 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:14.919 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:14.919 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:27:14.919 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:27:14.919 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:27:14.919 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:27:14.919 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:27:14.919 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.916+0000 7f563a00e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:14.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.928+0000 7f56327fc640 1 -- 172.21.15.155:0/2629126383 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5624001860 con 0x7f56340a98f0 2024-03-25T18:27:14.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.928+0000 7f56327fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:14.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.928+0000 7f56327fc640 1 -- 172.21.15.155:0/2629126383 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f5624001cb0 con 0x7f56340a98f0 2024-03-25T18:27:14.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.928+0000 7f563a00e640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:14.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.928+0000 7f563a00e640 4 set_mon_vals no callback set 2024-03-25T18:27:14.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.928+0000 7f56327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:14.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.928+0000 7f56327fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:14.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.928+0000 7f56327fc640 10 monclient: dump: 2024-03-25T18:27:14.931 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:27:14.932 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:27:14.933 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:14.933 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:14.933 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:14.933 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:27:14.933 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:27:14.933 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:27:14.933 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:27:14.933 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:27:14.933 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.929+0000 7f563a00e640 10 monclient: shutdown 2024-03-25T18:27:14.933 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.929+0000 7f563a00e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:14.933 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.929+0000 7f563a00e640 1 -- 172.21.15.155:0/2629126383 >> v1:172.21.15.155:6789/0 conn(0x7f56340a98f0 legacy=0x7f56340a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:14.933 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.929+0000 7f563a00e640 1 -- 172.21.15.155:0/2629126383 shutdown_connections 2024-03-25T18:27:14.933 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.929+0000 7f563a00e640 1 -- 172.21.15.155:0/2629126383 >> 172.21.15.155:0/2629126383 conn(0x7f56340b7ea0 msgr2=0x7f56340b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:14.933 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.929+0000 7f563a00e640 1 -- 172.21.15.155:0/2629126383 shutdown_connections 2024-03-25T18:27:14.934 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.929+0000 7f563a00e640 1 -- 172.21.15.155:0/2629126383 wait complete. 2024-03-25T18:27:14.935 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.930+0000 7f563a00e640 10 monclient: build_initial_monmap 2024-03-25T18:27:14.935 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.930+0000 7f563a00e640 10 monclient: monmap: 2024-03-25T18:27:14.935 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:27:14.935 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:14.935 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:27:14.936 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:27:14.936 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:14.936 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:27:14.936 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:27:14.936 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:27:14.936 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:27:14.936 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:27:14.936 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.930+0000 7f563a00e640 1 librados: starting msgr at 2024-03-25T18:27:14.936 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.930+0000 7f563a00e640 1 librados: starting objecter 2024-03-25T18:27:14.936 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.930+0000 7f563a00e640 1 Processor -- start 2024-03-25T18:27:14.936 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.931+0000 7f563a00e640 1 -- start start 2024-03-25T18:27:14.936 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.931+0000 7f563a00e640 1 librados: setting wanted keys 2024-03-25T18:27:14.936 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.931+0000 7f563a00e640 1 librados: calling monclient init 2024-03-25T18:27:14.936 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.931+0000 7f563a00e640 10 monclient: init 2024-03-25T18:27:14.937 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.931+0000 7f563a00e640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:14.937 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.931+0000 7f563a00e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:27:14.937 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.931+0000 7f563a00e640 10 monclient(hunting): picked mon.noname-b con 0x7f56340a98f0 addr v1:172.21.15.167:6789/0 2024-03-25T18:27:14.937 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.931+0000 7f563a00e640 10 monclient(hunting): picked mon.noname-a con 0x7f563413f710 addr v1:172.21.15.155:6789/0 2024-03-25T18:27:14.937 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.931+0000 7f563a00e640 10 monclient(hunting): picked mon.noname-c con 0x7f56341430c0 addr v1:172.21.15.155:6790/0 2024-03-25T18:27:14.937 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.931+0000 7f563a00e640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5634137580 con 0x7f563413f710 2024-03-25T18:27:14.937 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.931+0000 7f563a00e640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5634139020 con 0x7f56340a98f0 2024-03-25T18:27:14.937 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.931+0000 7f563a00e640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56341339d0 con 0x7f56341430c0 2024-03-25T18:27:14.937 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.931+0000 7f563a00e640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:14.937 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.931+0000 7f563a00e640 10 monclient(hunting): authenticate will time out at 7681.241528s 2024-03-25T18:27:14.938 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.937+0000 7f56337fe640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f56340a98f0 0x7f56341dabe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:42832/0 (socket says 172.21.15.155:42832) 2024-03-25T18:27:14.938 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.937+0000 7f56337fe640 1 -- 172.21.15.155:0/2591941224 learned_addr learned my addr 172.21.15.155:0/2591941224 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:27:14.938 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.938+0000 7f562bfff640 10 client.?.objecter ms_handle_connect 0x7f56340a98f0 2024-03-25T18:27:14.939 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.938+0000 7f562bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:14.947 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.946+0000 7f562bfff640 1 -- 172.21.15.155:0/2591941224 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1068557775 0 0) 0x7f5634139020 con 0x7f56340a98f0 2024-03-25T18:27:14.947 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.946+0000 7f562bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:27:14.947 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.946+0000 7f562bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:14.948 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.946+0000 7f562bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:14.948 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.946+0000 7f562bfff640 10 monclient(hunting): my global_id is 4327 2024-03-25T18:27:14.948 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.946+0000 7f562bfff640 1 -- 172.21.15.155:0/2591941224 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5614004eb0 con 0x7f56340a98f0 2024-03-25T18:27:14.948 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.947+0000 7f562bfff640 10 client.?.objecter ms_handle_connect 0x7f56341430c0 2024-03-25T18:27:14.948 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.947+0000 7f562bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:14.951 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.950+0000 7f562bfff640 10 client.?.objecter ms_handle_connect 0x7f563413f710 2024-03-25T18:27:14.951 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.950+0000 7f562bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:14.957 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.956+0000 7f562bfff640 1 -- 172.21.15.155:0/2591941224 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1269189733 0 0) 0x7f56341339d0 con 0x7f56341430c0 2024-03-25T18:27:14.957 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.956+0000 7f562bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:27:14.957 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.956+0000 7f562bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:14.957 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.956+0000 7f562bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:14.957 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.956+0000 7f562bfff640 10 monclient(hunting): my global_id is 4340 2024-03-25T18:27:14.958 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.956+0000 7f562bfff640 1 -- 172.21.15.155:0/2591941224 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5634139020 con 0x7f56341430c0 2024-03-25T18:27:14.959 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.958+0000 7f562bfff640 1 -- 172.21.15.155:0/2591941224 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1514390601 0 0) 0x7f5614004eb0 con 0x7f56340a98f0 2024-03-25T18:27:14.960 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.958+0000 7f562bfff640 1 -- 172.21.15.155:0/2591941224 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f56341339d0 con 0x7f56340a98f0 2024-03-25T18:27:14.960 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.958+0000 7f562bfff640 1 -- 172.21.15.155:0/2591941224 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f5624001b70 con 0x7f56340a98f0 2024-03-25T18:27:14.960 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.958+0000 7f562bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:14.960 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.958+0000 7f562bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:27:14.960 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.958+0000 7f562bfff640 10 monclient(hunting): dump: 2024-03-25T18:27:14.960 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:27:14.960 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:27:14.960 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:14.960 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:14.960 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:14.960 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:27:14.960 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:27:14.960 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:27:14.961 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:27:14.961 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:27:14.961 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.958+0000 7f562bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:14.970 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.968+0000 7f562bfff640 1 -- 172.21.15.155:0/2591941224 <== mon.1 v1:172.21.15.167:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2406220753 0 0) 0x7f56341339d0 con 0x7f56340a98f0 2024-03-25T18:27:14.971 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.968+0000 7f562bfff640 1 -- 172.21.15.155:0/2591941224 >> v1:172.21.15.155:6790/0 conn(0x7f56341430c0 legacy=0x7f56341332c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:14.971 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.968+0000 7f562bfff640 1 -- 172.21.15.155:0/2591941224 >> v1:172.21.15.155:6789/0 conn(0x7f563413f710 legacy=0x7f5634132bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:14.971 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.968+0000 7f562bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:27:14.971 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.968+0000 7f562bfff640 1 monclient: found mon.b 2024-03-25T18:27:14.971 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.968+0000 7f562bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:14.971 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.968+0000 7f562bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:27:14.971 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.968+0000 7f562bfff640 1 -- 172.21.15.155:0/2591941224 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56341db2f0 con 0x7f56340a98f0 2024-03-25T18:27:14.972 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.968+0000 7f562bfff640 10 monclient: _finish_auth 0 2024-03-25T18:27:14.972 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.968+0000 7f562bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:27:14.972 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.968+0000 7f562bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:14.972 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.968+0000 7f563a00e640 5 monclient: authenticate success, global_id 4327 2024-03-25T18:27:14.972 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.968+0000 7f563a00e640 10 monclient: _renew_subs 2024-03-25T18:27:14.972 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.968+0000 7f563a00e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:27:14.972 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.968+0000 7f563a00e640 1 -- 172.21.15.155:0/2591941224 --> v1:172.21.15.167:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5634137a10 con 0x7f56340a98f0 2024-03-25T18:27:14.972 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.968+0000 7f563a00e640 10 client.4327.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:14.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.968+0000 7f563a00e640 10 monclient: _renew_subs 2024-03-25T18:27:14.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.968+0000 7f563a00e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:27:14.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.968+0000 7f563a00e640 1 -- 172.21.15.155:0/2591941224 --> v1:172.21.15.167:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f56341380e0 con 0x7f56340a98f0 2024-03-25T18:27:14.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.968+0000 7f563a00e640 1 librados: init done 2024-03-25T18:27:14.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.969+0000 7f563a00e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:14.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.969+0000 7f563a00e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:14.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.969+0000 7f563a00e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:27:14.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:14.969+0000 7f563a00e640 1 -- 172.21.15.155:0/2591941224 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56340627b0 con 0x7f56340a98f0 2024-03-25T18:27:15.187 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:15.186+0000 7f562bfff640 1 -- 172.21.15.155:0/2591941224 <== mon.1 v1:172.21.15.167:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5624004bf0 con 0x7f56340a98f0 2024-03-25T18:27:15.187 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:15.186+0000 7f562bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:15.187 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:15.186+0000 7f5631ffb640 4 set_mon_vals no callback set 2024-03-25T18:27:15.254 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:15.253+0000 7f562bfff640 1 -- 172.21.15.155:0/2591941224 <== mon.1 v1:172.21.15.167:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f5624004f00 con 0x7f56340a98f0 2024-03-25T18:27:15.254 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:15.253+0000 7f562bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:15.254 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:15.253+0000 7f562bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:15.254 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:15.253+0000 7f562bfff640 10 monclient: dump: 2024-03-25T18:27:15.254 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:27:15.254 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:27:15.254 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:15.254 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:15.254 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:15.254 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:27:15.254 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:27:15.255 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:27:15.255 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:27:15.255 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:27:15.255 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:15.254+0000 7f562bfff640 1 -- 172.21.15.155:0/2591941224 <== mon.1 v1:172.21.15.167:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7f562401aed0 con 0x7f56340a98f0 2024-03-25T18:27:15.255 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:15.254+0000 7f5632ffd640 10 monclient: get_auth_request con 0x7f561406dcc0 auth_method 0 2024-03-25T18:27:15.255 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:15.255+0000 7f562bfff640 10 client.4327.objecter ms_handle_connect 0x7f561406dcc0 2024-03-25T18:27:15.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:15.259+0000 7f562bfff640 1 -- 172.21.15.155:0/2591941224 <== mon.1 v1:172.21.15.167:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7f56240549d0 con 0x7f56340a98f0 2024-03-25T18:27:15.260 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:15.259+0000 7f562bfff640 10 client.4327.objecter ms_dispatch 0x7f5634000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:15.260 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:15.259+0000 7f562bfff640 3 client.4327.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:15.260 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:15.259+0000 7f562bfff640 3 client.4327.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:15.260 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:15.259+0000 7f562bfff640 20 client.4327.objecter dump_active .. 0 homeless 2024-03-25T18:27:15.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:15.262+0000 7f562bfff640 1 -- 172.21.15.155:0/2591941224 <== mon.1 v1:172.21.15.167:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f56240504e0 con 0x7f56340a98f0 2024-03-25T18:27:15.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:15.262+0000 7f562bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:15.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:15.262+0000 7f562bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:15.478 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:15.477+0000 7f563a00e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:15.478 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:15.477+0000 7f563a00e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:15.478 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:15.477+0000 7f563a00e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:27:15.478 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:15.477+0000 7f563a00e640 1 -- 172.21.15.155:0/2591941224 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5634141850 con 0x7f56340a98f0 2024-03-25T18:27:15.502 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:15.501+0000 7f562bfff640 1 -- 172.21.15.155:0/2591941224 <== mon.1 v1:172.21.15.167:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7f5624052410 con 0x7f56340a98f0 2024-03-25T18:27:15.502 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:15.501+0000 7f562bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:15.502 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:15.501+0000 7f562bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:15.502 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:27:15.502 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:15.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:15.503+0000 7f563a00e640 10 librados: watch_flush enter 2024-03-25T18:27:15.505 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:15.503+0000 7f563a00e640 10 librados: watch_flush exit 2024-03-25T18:27:15.505 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:15.503+0000 7f563a00e640 20 client.4327.objecter shutdown clearing up homeless session... 2024-03-25T18:27:15.505 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:15.503+0000 7f563a00e640 10 client.4327.objecter successfully canceled tick 2024-03-25T18:27:15.505 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:15.503+0000 7f563a00e640 1 -- 172.21.15.155:0/2591941224 >> v1:172.21.15.167:6800/2919070503 conn(0x7f561406dcc0 legacy=0x7f5614070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:15.505 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:15.503+0000 7f563a00e640 10 monclient: shutdown 2024-03-25T18:27:15.505 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:15.503+0000 7f563a00e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:15.505 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:15.503+0000 7f563a00e640 1 -- 172.21.15.155:0/2591941224 >> v1:172.21.15.167:6789/0 conn(0x7f56340a98f0 legacy=0x7f56341dabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:15.505 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:15.503+0000 7f563a00e640 1 -- 172.21.15.155:0/2591941224 shutdown_connections 2024-03-25T18:27:15.505 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:15.503+0000 7f563a00e640 1 -- 172.21.15.155:0/2591941224 >> 172.21.15.155:0/2591941224 conn(0x7f56340b7ea0 msgr2=0x7f5634074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:15.505 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:15.503+0000 7f563a00e640 1 -- 172.21.15.155:0/2591941224 shutdown_connections 2024-03-25T18:27:15.505 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:15.503+0000 7f563a00e640 1 -- 172.21.15.155:0/2591941224 wait complete. 2024-03-25T18:27:15.505 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:15.503+0000 7f563a00e640 1 librados: shutdown 2024-03-25T18:27:15.515 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:20.782 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:02:54.387 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:20.789 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:02:54.394 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:20.820 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:02:54.425 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:20.844 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:02:54.423 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:20.847 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:02:54.452 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:20.849 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:02:54.428 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:20.861 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:02:54.465 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:20.875 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:02:54.454 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:20.885 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:02:54.489 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:22.322 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:22.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.444+0000 7ffb30b3b640 10 monclient: get_monmap_and_config 2024-03-25T18:27:22.653 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.444+0000 7ffb30b3b640 10 monclient: build_initial_monmap 2024-03-25T18:27:22.653 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.444+0000 7ffb30b3b640 10 monclient: monmap: 2024-03-25T18:27:22.653 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:27:22.653 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:27:22.653 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:27:22.445426+0000 2024-03-25T18:27:22.654 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:27:22.445426+0000 2024-03-25T18:27:22.654 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:22.654 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:27:22.654 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:27:22.654 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:27:22.654 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:27:22.654 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:27:22.654 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.445+0000 7ffb30b3b640 1 Processor -- start 2024-03-25T18:27:22.654 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.445+0000 7ffb30b3b640 1 -- start start 2024-03-25T18:27:22.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.445+0000 7ffb30b3b640 10 monclient: init 2024-03-25T18:27:22.663 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.445+0000 7ffb30b3b640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:22.663 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.445+0000 7ffb30b3b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:27:22.663 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.445+0000 7ffb30b3b640 10 monclient(hunting): picked mon.noname-c con 0x7ffb2c0a98f0 addr v1:172.21.15.167:6789/0 2024-03-25T18:27:22.663 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.446+0000 7ffb30b3b640 10 monclient(hunting): picked mon.noname-a con 0x7ffb2c13f710 addr v1:172.21.15.155:6789/0 2024-03-25T18:27:22.663 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.446+0000 7ffb30b3b640 10 monclient(hunting): picked mon.noname-b con 0x7ffb2c1430c0 addr v1:172.21.15.155:6790/0 2024-03-25T18:27:22.663 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.446+0000 7ffb30b3b640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb2c142520 con 0x7ffb2c13f710 2024-03-25T18:27:22.663 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.446+0000 7ffb30b3b640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb2c0b28a0 con 0x7ffb2c0a98f0 2024-03-25T18:27:22.663 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.446+0000 7ffb30b3b640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb2c0b2a30 con 0x7ffb2c1430c0 2024-03-25T18:27:22.663 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.446+0000 7ffb30b3b640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:22.663 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.446+0000 7ffb30b3b640 10 monclient(hunting): authenticate will time out at 7688.755888s 2024-03-25T18:27:22.663 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.449+0000 7ffb2a59c640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7ffb2c0a98f0 0x7ffb2c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:60962/0 (socket says 172.21.15.155:60962) 2024-03-25T18:27:22.663 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.449+0000 7ffb2a59c640 1 -- 172.21.15.155:0/903325749 learned_addr learned my addr 172.21.15.155:0/903325749 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:27:22.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.457+0000 7ffb2959a640 1 -- 172.21.15.155:0/903325749 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3823267872 0 0) 0x7ffb2c0b28a0 con 0x7ffb2c0a98f0 2024-03-25T18:27:22.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.457+0000 7ffb2959a640 10 monclient(hunting): _negotiate 2024-03-25T18:27:22.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.457+0000 7ffb2959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:22.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.457+0000 7ffb2959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:22.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.457+0000 7ffb2959a640 10 monclient(hunting): my global_id is 4330 2024-03-25T18:27:22.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.457+0000 7ffb2959a640 1 -- 172.21.15.155:0/903325749 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb14003520 con 0x7ffb2c0a98f0 2024-03-25T18:27:22.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.463+0000 7ffb2959a640 1 -- 172.21.15.155:0/903325749 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4190895174 0 0) 0x7ffb14003520 con 0x7ffb2c0a98f0 2024-03-25T18:27:22.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.463+0000 7ffb2959a640 1 -- 172.21.15.155:0/903325749 >> v1:172.21.15.155:6790/0 conn(0x7ffb2c1430c0 legacy=0x7ffb2c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:22.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.463+0000 7ffb2959a640 1 -- 172.21.15.155:0/903325749 >> v1:172.21.15.155:6789/0 conn(0x7ffb2c13f710 legacy=0x7ffb2c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:27:22.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.463+0000 7ffb2959a640 10 monclient: _finish_hunting 0 2024-03-25T18:27:22.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.463+0000 7ffb2959a640 1 monclient: found mon.noname-c 2024-03-25T18:27:22.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.463+0000 7ffb2959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:22.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.463+0000 7ffb2959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.167:6789/0 2024-03-25T18:27:22.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.463+0000 7ffb2959a640 1 -- 172.21.15.155:0/903325749 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb2c0b44c0 con 0x7ffb2c0a98f0 2024-03-25T18:27:22.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.463+0000 7ffb2959a640 10 monclient: _finish_auth 0 2024-03-25T18:27:22.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.463+0000 7ffb2959a640 10 monclient: _check_auth_tickets 2024-03-25T18:27:22.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.463+0000 7ffb2959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:22.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.463+0000 7ffb2959a640 1 -- 172.21.15.155:0/903325749 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7ffb1c001690 con 0x7ffb2c0a98f0 2024-03-25T18:27:22.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.463+0000 7ffb30b3b640 5 monclient: authenticate success, global_id 4330 2024-03-25T18:27:22.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.463+0000 7ffb30b3b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:22.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.463+0000 7ffb2959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:22.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.463+0000 7ffb2959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:27:22.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.463+0000 7ffb2959a640 10 monclient: dump: 2024-03-25T18:27:22.665 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:27:22.665 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:27:22.665 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:22.666 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:22.666 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:22.666 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:27:22.666 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:27:22.666 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:27:22.666 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:27:22.666 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:27:22.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.463+0000 7ffb30b3b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:22.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.468+0000 7ffb2959a640 1 -- 172.21.15.155:0/903325749 <== mon.1 v1:172.21.15.167:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb1c001860 con 0x7ffb2c0a98f0 2024-03-25T18:27:22.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.468+0000 7ffb2959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:22.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.468+0000 7ffb2959a640 1 -- 172.21.15.155:0/903325749 <== mon.1 v1:172.21.15.167:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7ffb1c001cb0 con 0x7ffb2c0a98f0 2024-03-25T18:27:22.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.468+0000 7ffb2959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:22.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.468+0000 7ffb2959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:22.667 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.468+0000 7ffb2959a640 10 monclient: dump: 2024-03-25T18:27:22.667 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:27:22.667 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:27:22.667 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:22.667 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:22.667 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:22.667 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:27:22.667 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:27:22.667 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:27:22.667 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:27:22.667 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:27:22.667 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.468+0000 7ffb30b3b640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:22.667 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.468+0000 7ffb30b3b640 4 set_mon_vals no callback set 2024-03-25T18:27:22.667 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.469+0000 7ffb30b3b640 10 monclient: shutdown 2024-03-25T18:27:22.667 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.469+0000 7ffb30b3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:22.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.469+0000 7ffb30b3b640 1 -- 172.21.15.155:0/903325749 >> v1:172.21.15.167:6789/0 conn(0x7ffb2c0a98f0 legacy=0x7ffb2c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:22.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.469+0000 7ffb30b3b640 1 -- 172.21.15.155:0/903325749 shutdown_connections 2024-03-25T18:27:22.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.469+0000 7ffb30b3b640 1 -- 172.21.15.155:0/903325749 >> 172.21.15.155:0/903325749 conn(0x7ffb2c0b7ea0 msgr2=0x7ffb2c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:22.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.469+0000 7ffb30b3b640 1 -- 172.21.15.155:0/903325749 shutdown_connections 2024-03-25T18:27:22.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.469+0000 7ffb30b3b640 1 -- 172.21.15.155:0/903325749 wait complete. 2024-03-25T18:27:22.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.470+0000 7ffb30b3b640 10 monclient: build_initial_monmap 2024-03-25T18:27:22.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.470+0000 7ffb30b3b640 10 monclient: monmap: 2024-03-25T18:27:22.668 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:27:22.668 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:22.668 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:27:22.668 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:27:22.668 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:22.668 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:27:22.668 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:27:22.668 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:27:22.669 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:27:22.669 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:27:22.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.470+0000 7ffb30b3b640 1 librados: starting msgr at 2024-03-25T18:27:22.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.470+0000 7ffb30b3b640 1 librados: starting objecter 2024-03-25T18:27:22.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.471+0000 7ffb30b3b640 1 Processor -- start 2024-03-25T18:27:22.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.471+0000 7ffb30b3b640 1 -- start start 2024-03-25T18:27:22.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.471+0000 7ffb30b3b640 1 librados: setting wanted keys 2024-03-25T18:27:22.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.471+0000 7ffb30b3b640 1 librados: calling monclient init 2024-03-25T18:27:22.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.471+0000 7ffb30b3b640 10 monclient: init 2024-03-25T18:27:22.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.471+0000 7ffb30b3b640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:22.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.471+0000 7ffb30b3b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:27:22.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.471+0000 7ffb30b3b640 10 monclient(hunting): picked mon.noname-c con 0x7ffb2c0a98f0 addr v1:172.21.15.155:6790/0 2024-03-25T18:27:22.670 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.471+0000 7ffb30b3b640 10 monclient(hunting): picked mon.noname-a con 0x7ffb2c13f710 addr v1:172.21.15.155:6789/0 2024-03-25T18:27:22.670 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.471+0000 7ffb30b3b640 10 monclient(hunting): picked mon.noname-b con 0x7ffb2c1430c0 addr v1:172.21.15.167:6789/0 2024-03-25T18:27:22.670 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.471+0000 7ffb30b3b640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb2c137320 con 0x7ffb2c13f710 2024-03-25T18:27:22.670 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.471+0000 7ffb30b3b640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb2c1368a0 con 0x7ffb2c1430c0 2024-03-25T18:27:22.670 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.471+0000 7ffb30b3b640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb2c136a30 con 0x7ffb2c0a98f0 2024-03-25T18:27:22.670 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.471+0000 7ffb30b3b640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:22.670 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.471+0000 7ffb30b3b640 10 monclient(hunting): authenticate will time out at 7688.781647s 2024-03-25T18:27:22.670 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.487+0000 7ffb29d9b640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7ffb2c13f710 0x7ffb2c134f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:58952/0 (socket says 172.21.15.155:58952) 2024-03-25T18:27:22.670 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.487+0000 7ffb29d9b640 1 -- 172.21.15.155:0/2761953635 learned_addr learned my addr 172.21.15.155:0/2761953635 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:27:22.670 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.489+0000 7ffb1affd640 10 client.?.objecter ms_handle_connect 0x7ffb2c0a98f0 2024-03-25T18:27:22.670 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.489+0000 7ffb1affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:22.670 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.493+0000 7ffb1affd640 10 client.?.objecter ms_handle_connect 0x7ffb2c13f710 2024-03-25T18:27:22.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.493+0000 7ffb1affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:22.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.494+0000 7ffb1affd640 10 client.?.objecter ms_handle_connect 0x7ffb2c1430c0 2024-03-25T18:27:22.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.494+0000 7ffb1affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:22.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.503+0000 7ffb1affd640 1 -- 172.21.15.155:0/2761953635 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1897265056 0 0) 0x7ffb2c136a30 con 0x7ffb2c0a98f0 2024-03-25T18:27:22.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.503+0000 7ffb1affd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:22.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.503+0000 7ffb1affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:22.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.503+0000 7ffb1affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:22.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.503+0000 7ffb1affd640 10 monclient(hunting): my global_id is 4346 2024-03-25T18:27:22.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.503+0000 7ffb1affd640 1 -- 172.21.15.155:0/2761953635 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb0c004eb0 con 0x7ffb2c0a98f0 2024-03-25T18:27:22.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.507+0000 7ffb1affd640 1 -- 172.21.15.155:0/2761953635 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2991533605 0 0) 0x7ffb2c137320 con 0x7ffb2c13f710 2024-03-25T18:27:22.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.507+0000 7ffb1affd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:22.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.507+0000 7ffb1affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:22.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.507+0000 7ffb1affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:22.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.507+0000 7ffb1affd640 10 monclient(hunting): my global_id is 4338 2024-03-25T18:27:22.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.507+0000 7ffb1affd640 1 -- 172.21.15.155:0/2761953635 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb2c136a30 con 0x7ffb2c13f710 2024-03-25T18:27:22.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.515+0000 7ffb1affd640 1 -- 172.21.15.155:0/2761953635 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 277470989 0 0) 0x7ffb0c004eb0 con 0x7ffb2c0a98f0 2024-03-25T18:27:22.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.515+0000 7ffb1affd640 1 -- 172.21.15.155:0/2761953635 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb2c137320 con 0x7ffb2c0a98f0 2024-03-25T18:27:22.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.515+0000 7ffb1affd640 1 -- 172.21.15.155:0/2761953635 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7ffb1c001b70 con 0x7ffb2c0a98f0 2024-03-25T18:27:22.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.515+0000 7ffb1affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:22.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.515+0000 7ffb1affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:27:22.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.515+0000 7ffb1affd640 10 monclient(hunting): dump: 2024-03-25T18:27:22.672 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:27:22.672 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:27:22.672 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:22.672 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:22.672 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:22.672 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:27:22.672 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:27:22.673 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:27:22.673 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:27:22.673 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:27:22.673 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.515+0000 7ffb1affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:22.673 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.518+0000 7ffb1affd640 1 -- 172.21.15.155:0/2761953635 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 763978636 0 0) 0x7ffb2c1368a0 con 0x7ffb2c1430c0 2024-03-25T18:27:22.673 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.518+0000 7ffb1affd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:22.673 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.518+0000 7ffb1affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:22.673 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.518+0000 7ffb1affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:22.673 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.518+0000 7ffb1affd640 10 monclient(hunting): my global_id is 4333 2024-03-25T18:27:22.673 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.518+0000 7ffb1affd640 1 -- 172.21.15.155:0/2761953635 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb0c004eb0 con 0x7ffb2c1430c0 2024-03-25T18:27:22.673 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.535+0000 7ffb1affd640 1 -- 172.21.15.155:0/2761953635 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2061468909 0 0) 0x7ffb2c137320 con 0x7ffb2c0a98f0 2024-03-25T18:27:22.673 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.535+0000 7ffb1affd640 1 -- 172.21.15.155:0/2761953635 >> v1:172.21.15.167:6789/0 conn(0x7ffb2c1430c0 legacy=0x7ffb2c1e1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:22.673 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.535+0000 7ffb1affd640 1 -- 172.21.15.155:0/2761953635 >> v1:172.21.15.155:6789/0 conn(0x7ffb2c13f710 legacy=0x7ffb2c134f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:22.673 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.535+0000 7ffb1affd640 10 monclient: _finish_hunting 0 2024-03-25T18:27:22.674 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.535+0000 7ffb1affd640 1 monclient: found mon.c 2024-03-25T18:27:22.674 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.535+0000 7ffb1affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:22.674 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.535+0000 7ffb1affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:27:22.674 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.535+0000 7ffb1affd640 1 -- 172.21.15.155:0/2761953635 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb2c139690 con 0x7ffb2c0a98f0 2024-03-25T18:27:22.674 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.535+0000 7ffb1affd640 10 monclient: _finish_auth 0 2024-03-25T18:27:22.674 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.535+0000 7ffb1affd640 10 monclient: _check_auth_tickets 2024-03-25T18:27:22.674 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.535+0000 7ffb1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:22.674 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.535+0000 7ffb30b3b640 5 monclient: authenticate success, global_id 4346 2024-03-25T18:27:22.674 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.535+0000 7ffb30b3b640 10 monclient: _renew_subs 2024-03-25T18:27:22.674 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.535+0000 7ffb30b3b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:27:22.674 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.535+0000 7ffb30b3b640 1 -- 172.21.15.155:0/2761953635 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb2c139940 con 0x7ffb2c0a98f0 2024-03-25T18:27:22.674 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.535+0000 7ffb30b3b640 10 client.4346.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:22.674 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.535+0000 7ffb30b3b640 10 monclient: _renew_subs 2024-03-25T18:27:22.675 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.535+0000 7ffb30b3b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:27:22.675 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.535+0000 7ffb30b3b640 1 -- 172.21.15.155:0/2761953635 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb2c137990 con 0x7ffb2c0a98f0 2024-03-25T18:27:22.675 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.535+0000 7ffb30b3b640 1 librados: init done 2024-03-25T18:27:22.675 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.536+0000 7ffb30b3b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:22.675 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.536+0000 7ffb30b3b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:22.675 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.536+0000 7ffb30b3b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:27:22.675 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.536+0000 7ffb30b3b640 1 -- 172.21.15.155:0/2761953635 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb2c0b3ef0 con 0x7ffb2c0a98f0 2024-03-25T18:27:22.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.845+0000 7ffb1affd640 1 -- 172.21.15.155:0/2761953635 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb1c001010 con 0x7ffb2c0a98f0 2024-03-25T18:27:22.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.845+0000 7ffb1affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:22.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.845+0000 7ffb1affd640 1 -- 172.21.15.155:0/2761953635 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7ffb1c004bf0 con 0x7ffb2c0a98f0 2024-03-25T18:27:22.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.845+0000 7ffb1affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:22.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.845+0000 7ffb1affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:22.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.845+0000 7ffb1affd640 10 monclient: dump: 2024-03-25T18:27:22.847 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:27:22.847 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:27:22.847 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:22.847 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:22.847 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:22.847 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:27:22.847 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:27:22.847 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:27:22.847 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:27:22.847 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:27:22.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.845+0000 7ffb28d99640 4 set_mon_vals no callback set 2024-03-25T18:27:22.848 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.845+0000 7ffb1affd640 1 -- 172.21.15.155:0/2761953635 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7ffb1c01abc0 con 0x7ffb2c0a98f0 2024-03-25T18:27:22.848 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.846+0000 7ffb29d9b640 10 monclient: get_auth_request con 0x7ffb0c06dd10 auth_method 0 2024-03-25T18:27:22.848 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.846+0000 7ffb1affd640 10 client.4346.objecter ms_handle_connect 0x7ffb0c06dd10 2024-03-25T18:27:22.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.852+0000 7ffb1affd640 1 -- 172.21.15.155:0/2761953635 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7ffb1c054910 con 0x7ffb2c0a98f0 2024-03-25T18:27:22.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.852+0000 7ffb1affd640 10 client.4346.objecter ms_dispatch 0x7ffb2c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:22.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.852+0000 7ffb1affd640 3 client.4346.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:22.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.852+0000 7ffb1affd640 3 client.4346.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:22.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.852+0000 7ffb1affd640 20 client.4346.objecter dump_active .. 0 homeless 2024-03-25T18:27:22.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.852+0000 7ffb1affd640 1 -- 172.21.15.155:0/2761953635 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffb1c050250 con 0x7ffb2c0a98f0 2024-03-25T18:27:22.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.853+0000 7ffb1affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:22.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:22.853+0000 7ffb1affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:23.109 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:23.108+0000 7ffb30b3b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:23.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:23.108+0000 7ffb30b3b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:23.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:23.108+0000 7ffb30b3b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:27:23.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:23.108+0000 7ffb30b3b640 1 -- 172.21.15.155:0/2761953635 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffb2c139140 con 0x7ffb2c0a98f0 2024-03-25T18:27:23.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:23.139+0000 7ffb1affd640 1 -- 172.21.15.155:0/2761953635 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7ffb2c139140 con 0x7ffb2c0a98f0 2024-03-25T18:27:23.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:23.139+0000 7ffb1affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:23.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:23.139+0000 7ffb1affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:23.140 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:27:23.140 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:23.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:23.141+0000 7ffb30b3b640 10 librados: watch_flush enter 2024-03-25T18:27:23.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:23.141+0000 7ffb30b3b640 10 librados: watch_flush exit 2024-03-25T18:27:23.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:23.141+0000 7ffb30b3b640 20 client.4346.objecter shutdown clearing up homeless session... 2024-03-25T18:27:23.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:23.141+0000 7ffb30b3b640 10 client.4346.objecter successfully canceled tick 2024-03-25T18:27:23.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:23.141+0000 7ffb30b3b640 1 -- 172.21.15.155:0/2761953635 >> v1:172.21.15.167:6800/2919070503 conn(0x7ffb0c06dd10 legacy=0x7ffb0c0701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:23.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:23.141+0000 7ffb30b3b640 10 monclient: shutdown 2024-03-25T18:27:23.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:23.141+0000 7ffb30b3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:23.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:23.141+0000 7ffb30b3b640 1 -- 172.21.15.155:0/2761953635 >> v1:172.21.15.155:6790/0 conn(0x7ffb2c0a98f0 legacy=0x7ffb2c13b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:23.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:23.141+0000 7ffb30b3b640 1 -- 172.21.15.155:0/2761953635 shutdown_connections 2024-03-25T18:27:23.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:23.141+0000 7ffb30b3b640 1 -- 172.21.15.155:0/2761953635 >> 172.21.15.155:0/2761953635 conn(0x7ffb2c0b7ea0 msgr2=0x7ffb2c13c4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:23.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:23.141+0000 7ffb30b3b640 1 -- 172.21.15.155:0/2761953635 shutdown_connections 2024-03-25T18:27:23.144 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:23.141+0000 7ffb30b3b640 1 -- 172.21.15.155:0/2761953635 wait complete. 2024-03-25T18:27:23.144 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:23.141+0000 7ffb30b3b640 1 librados: shutdown 2024-03-25T18:27:23.153 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:29.961 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:30.075 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.073+0000 7f1b94720640 10 monclient: get_monmap_and_config 2024-03-25T18:27:30.075 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.073+0000 7f1b94720640 10 monclient: build_initial_monmap 2024-03-25T18:27:30.075 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.073+0000 7f1b94720640 10 monclient: monmap: 2024-03-25T18:27:30.075 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:27:30.075 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:27:30.075 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:27:30.074795+0000 2024-03-25T18:27:30.075 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:27:30.074795+0000 2024-03-25T18:27:30.075 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:30.075 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:27:30.076 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:27:30.076 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:27:30.076 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:27:30.076 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:27:30.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.074+0000 7f1b94720640 1 Processor -- start 2024-03-25T18:27:30.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.074+0000 7f1b94720640 1 -- start start 2024-03-25T18:27:30.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.074+0000 7f1b94720640 10 monclient: init 2024-03-25T18:27:30.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.075+0000 7f1b94720640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:30.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.075+0000 7f1b94720640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:27:30.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.075+0000 7f1b94720640 10 monclient(hunting): picked mon.noname-a con 0x7f1b8c0a98f0 addr v1:172.21.15.155:6789/0 2024-03-25T18:27:30.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.075+0000 7f1b94720640 10 monclient(hunting): picked mon.noname-c con 0x7f1b8c13f710 addr v1:172.21.15.167:6789/0 2024-03-25T18:27:30.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.075+0000 7f1b94720640 10 monclient(hunting): picked mon.noname-b con 0x7f1b8c1430c0 addr v1:172.21.15.155:6790/0 2024-03-25T18:27:30.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.075+0000 7f1b94720640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b8c142520 con 0x7f1b8c0a98f0 2024-03-25T18:27:30.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.075+0000 7f1b94720640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b8c0b28a0 con 0x7f1b8c13f710 2024-03-25T18:27:30.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.075+0000 7f1b94720640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b8c0b2a30 con 0x7f1b8c1430c0 2024-03-25T18:27:30.078 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.075+0000 7f1b94720640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:30.078 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.075+0000 7f1b94720640 10 monclient(hunting): authenticate will time out at 7696.385353s 2024-03-25T18:27:30.078 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.077+0000 7f1b91cbb640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f1b8c13f710 0x7f1b8c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:54578/0 (socket says 172.21.15.155:54578) 2024-03-25T18:27:30.078 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.077+0000 7f1b91cbb640 1 -- 172.21.15.155:0/2821364328 learned_addr learned my addr 172.21.15.155:0/2821364328 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:27:30.083 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.082+0000 7f1b914ba640 1 -- 172.21.15.155:0/2821364328 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1165184161 0 0) 0x7f1b8c0b28a0 con 0x7f1b8c13f710 2024-03-25T18:27:30.083 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.082+0000 7f1b914ba640 10 monclient(hunting): _negotiate 2024-03-25T18:27:30.083 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.082+0000 7f1b914ba640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:30.083 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.082+0000 7f1b914ba640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:30.083 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.082+0000 7f1b914ba640 10 monclient(hunting): my global_id is 4336 2024-03-25T18:27:30.083 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.082+0000 7f1b914ba640 1 -- 172.21.15.155:0/2821364328 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b78003520 con 0x7f1b8c13f710 2024-03-25T18:27:30.084 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.084+0000 7f1b914ba640 1 -- 172.21.15.155:0/2821364328 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3168538248 0 0) 0x7f1b8c142520 con 0x7f1b8c0a98f0 2024-03-25T18:27:30.085 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.084+0000 7f1b914ba640 10 monclient(hunting): _negotiate 2024-03-25T18:27:30.085 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.084+0000 7f1b914ba640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:30.085 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.084+0000 7f1b914ba640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:30.085 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.084+0000 7f1b914ba640 10 monclient(hunting): my global_id is 4341 2024-03-25T18:27:30.085 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.084+0000 7f1b914ba640 1 -- 172.21.15.155:0/2821364328 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b8c0b28a0 con 0x7f1b8c0a98f0 2024-03-25T18:27:30.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.102+0000 7f1b914ba640 1 -- 172.21.15.155:0/2821364328 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 30925140 0 0) 0x7f1b78003520 con 0x7f1b8c13f710 2024-03-25T18:27:30.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.102+0000 7f1b914ba640 1 -- 172.21.15.155:0/2821364328 >> v1:172.21.15.155:6790/0 conn(0x7f1b8c1430c0 legacy=0x7f1b8c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:30.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.102+0000 7f1b914ba640 1 -- 172.21.15.155:0/2821364328 >> v1:172.21.15.155:6789/0 conn(0x7f1b8c0a98f0 legacy=0x7f1b8c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:30.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.102+0000 7f1b914ba640 10 monclient: _finish_hunting 0 2024-03-25T18:27:30.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.102+0000 7f1b914ba640 1 monclient: found mon.noname-c 2024-03-25T18:27:30.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.102+0000 7f1b914ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:30.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.102+0000 7f1b914ba640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.167:6789/0 2024-03-25T18:27:30.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.102+0000 7f1b914ba640 1 -- 172.21.15.155:0/2821364328 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b8c0b44c0 con 0x7f1b8c13f710 2024-03-25T18:27:30.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.102+0000 7f1b914ba640 10 monclient: _finish_auth 0 2024-03-25T18:27:30.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.102+0000 7f1b914ba640 10 monclient: _check_auth_tickets 2024-03-25T18:27:30.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.102+0000 7f1b914ba640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:30.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.102+0000 7f1b914ba640 1 -- 172.21.15.155:0/2821364328 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f1b7c001690 con 0x7f1b8c13f710 2024-03-25T18:27:30.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.102+0000 7f1b914ba640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:30.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.102+0000 7f1b914ba640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:27:30.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.102+0000 7f1b914ba640 10 monclient: dump: 2024-03-25T18:27:30.104 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:27:30.104 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:27:30.104 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:30.104 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:30.105 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:30.105 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:27:30.105 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:27:30.105 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:27:30.105 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:27:30.105 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:27:30.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.102+0000 7f1b94720640 5 monclient: authenticate success, global_id 4336 2024-03-25T18:27:30.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.102+0000 7f1b94720640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:30.112 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.111+0000 7f1b914ba640 1 -- 172.21.15.155:0/2821364328 <== mon.1 v1:172.21.15.167:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b7c0017f0 con 0x7f1b8c13f710 2024-03-25T18:27:30.112 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.111+0000 7f1b914ba640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:30.112 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.111+0000 7f1b914ba640 1 -- 172.21.15.155:0/2821364328 <== mon.1 v1:172.21.15.167:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f1b7c001c00 con 0x7f1b8c13f710 2024-03-25T18:27:30.112 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.111+0000 7f1b914ba640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:30.112 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.111+0000 7f1b914ba640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:30.112 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.111+0000 7f1b914ba640 10 monclient: dump: 2024-03-25T18:27:30.112 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:27:30.112 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:27:30.112 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:30.112 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:30.112 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:30.113 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:27:30.113 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:27:30.113 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:27:30.113 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:27:30.113 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:27:30.113 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.111+0000 7f1b94720640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:30.113 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.111+0000 7f1b94720640 4 set_mon_vals no callback set 2024-03-25T18:27:30.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.112+0000 7f1b94720640 10 monclient: shutdown 2024-03-25T18:27:30.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.112+0000 7f1b94720640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:30.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.112+0000 7f1b94720640 1 -- 172.21.15.155:0/2821364328 >> v1:172.21.15.167:6789/0 conn(0x7f1b8c13f710 legacy=0x7f1b8c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:30.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.112+0000 7f1b94720640 1 -- 172.21.15.155:0/2821364328 shutdown_connections 2024-03-25T18:27:30.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.112+0000 7f1b94720640 1 -- 172.21.15.155:0/2821364328 >> 172.21.15.155:0/2821364328 conn(0x7f1b8c0b7ea0 msgr2=0x7f1b8c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:30.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.112+0000 7f1b94720640 1 -- 172.21.15.155:0/2821364328 shutdown_connections 2024-03-25T18:27:30.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.112+0000 7f1b94720640 1 -- 172.21.15.155:0/2821364328 wait complete. 2024-03-25T18:27:30.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.113+0000 7f1b94720640 10 monclient: build_initial_monmap 2024-03-25T18:27:30.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.113+0000 7f1b94720640 10 monclient: monmap: 2024-03-25T18:27:30.116 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:27:30.116 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:30.116 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:27:30.116 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:27:30.116 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:30.116 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:27:30.116 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:27:30.116 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:27:30.116 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:27:30.116 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:27:30.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.113+0000 7f1b94720640 1 librados: starting msgr at 2024-03-25T18:27:30.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.113+0000 7f1b94720640 1 librados: starting objecter 2024-03-25T18:27:30.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.113+0000 7f1b94720640 1 Processor -- start 2024-03-25T18:27:30.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.113+0000 7f1b94720640 1 -- start start 2024-03-25T18:27:30.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.113+0000 7f1b94720640 1 librados: setting wanted keys 2024-03-25T18:27:30.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.113+0000 7f1b94720640 1 librados: calling monclient init 2024-03-25T18:27:30.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.113+0000 7f1b94720640 10 monclient: init 2024-03-25T18:27:30.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.114+0000 7f1b94720640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:30.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.114+0000 7f1b94720640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:27:30.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.114+0000 7f1b94720640 10 monclient(hunting): picked mon.noname-a con 0x7f1b8c0a98f0 addr v1:172.21.15.155:6789/0 2024-03-25T18:27:30.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.114+0000 7f1b94720640 10 monclient(hunting): picked mon.noname-b con 0x7f1b8c13f710 addr v1:172.21.15.167:6789/0 2024-03-25T18:27:30.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.114+0000 7f1b94720640 10 monclient(hunting): picked mon.noname-c con 0x7f1b8c1430c0 addr v1:172.21.15.155:6790/0 2024-03-25T18:27:30.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.114+0000 7f1b94720640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b8c138a70 con 0x7f1b8c0a98f0 2024-03-25T18:27:30.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.114+0000 7f1b94720640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b8c0aba50 con 0x7f1b8c13f710 2024-03-25T18:27:30.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.114+0000 7f1b94720640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b8c1d77a0 con 0x7f1b8c1430c0 2024-03-25T18:27:30.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.114+0000 7f1b94720640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:30.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.114+0000 7f1b94720640 10 monclient(hunting): authenticate will time out at 7696.424389s 2024-03-25T18:27:30.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.122+0000 7f1b91cbb640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f1b8c13f710 0x7f1b8c1d7cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:54638/0 (socket says 172.21.15.155:54638) 2024-03-25T18:27:30.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.122+0000 7f1b91cbb640 1 -- 172.21.15.155:0/1667209371 learned_addr learned my addr 172.21.15.155:0/1667209371 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:27:30.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.123+0000 7f1b82ffd640 10 client.?.objecter ms_handle_connect 0x7f1b8c13f710 2024-03-25T18:27:30.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.123+0000 7f1b82ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:30.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.124+0000 7f1b82ffd640 10 client.?.objecter ms_handle_connect 0x7f1b8c1430c0 2024-03-25T18:27:30.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.124+0000 7f1b82ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:30.126 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.125+0000 7f1b82ffd640 10 client.?.objecter ms_handle_connect 0x7f1b8c0a98f0 2024-03-25T18:27:30.126 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.125+0000 7f1b82ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:30.129 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.128+0000 7f1b82ffd640 1 -- 172.21.15.155:0/1667209371 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1407946792 0 0) 0x7f1b8c0aba50 con 0x7f1b8c13f710 2024-03-25T18:27:30.129 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.128+0000 7f1b82ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:30.129 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.128+0000 7f1b82ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:30.129 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.128+0000 7f1b82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:30.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.128+0000 7f1b82ffd640 10 monclient(hunting): my global_id is 4339 2024-03-25T18:27:30.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.128+0000 7f1b82ffd640 1 -- 172.21.15.155:0/1667209371 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b70004eb0 con 0x7f1b8c13f710 2024-03-25T18:27:30.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.142+0000 7f1b82ffd640 1 -- 172.21.15.155:0/1667209371 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 681612018 0 0) 0x7f1b8c138a70 con 0x7f1b8c0a98f0 2024-03-25T18:27:30.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.142+0000 7f1b82ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:30.144 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.142+0000 7f1b82ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:30.144 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.142+0000 7f1b82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:30.144 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.142+0000 7f1b82ffd640 10 monclient(hunting): my global_id is 4344 2024-03-25T18:27:30.144 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.142+0000 7f1b82ffd640 1 -- 172.21.15.155:0/1667209371 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b8c0aba50 con 0x7f1b8c0a98f0 2024-03-25T18:27:30.149 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.148+0000 7f1b82ffd640 1 -- 172.21.15.155:0/1667209371 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 636753986 0 0) 0x7f1b70004eb0 con 0x7f1b8c13f710 2024-03-25T18:27:30.149 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.148+0000 7f1b82ffd640 1 -- 172.21.15.155:0/1667209371 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b8c138a70 con 0x7f1b8c13f710 2024-03-25T18:27:30.150 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.148+0000 7f1b82ffd640 1 -- 172.21.15.155:0/1667209371 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f1b7c001bc0 con 0x7f1b8c13f710 2024-03-25T18:27:30.150 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.148+0000 7f1b82ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:30.150 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.148+0000 7f1b82ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:27:30.150 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.148+0000 7f1b82ffd640 10 monclient(hunting): dump: 2024-03-25T18:27:30.150 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:27:30.150 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:27:30.150 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:30.151 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:30.151 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:30.151 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:27:30.151 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:27:30.151 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:27:30.151 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:27:30.151 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:27:30.151 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.148+0000 7f1b82ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:30.153 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.152+0000 7f1b82ffd640 1 -- 172.21.15.155:0/1667209371 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 296668084 0 0) 0x7f1b8c1d77a0 con 0x7f1b8c1430c0 2024-03-25T18:27:30.154 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.152+0000 7f1b82ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:30.154 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.152+0000 7f1b82ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:30.154 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.152+0000 7f1b82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:30.154 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.152+0000 7f1b82ffd640 10 monclient(hunting): my global_id is 4352 2024-03-25T18:27:30.154 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.152+0000 7f1b82ffd640 1 -- 172.21.15.155:0/1667209371 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b70004eb0 con 0x7f1b8c1430c0 2024-03-25T18:27:30.163 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.160+0000 7f1b82ffd640 1 -- 172.21.15.155:0/1667209371 <== mon.1 v1:172.21.15.167:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1478520714 0 0) 0x7f1b8c138a70 con 0x7f1b8c13f710 2024-03-25T18:27:30.163 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.160+0000 7f1b82ffd640 1 -- 172.21.15.155:0/1667209371 >> v1:172.21.15.155:6790/0 conn(0x7f1b8c1430c0 legacy=0x7f1b8c1d4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:30.163 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.160+0000 7f1b82ffd640 1 -- 172.21.15.155:0/1667209371 >> v1:172.21.15.155:6789/0 conn(0x7f1b8c0a98f0 legacy=0x7f1b8c138360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:30.163 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.160+0000 7f1b82ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:27:30.163 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.160+0000 7f1b82ffd640 1 monclient: found mon.b 2024-03-25T18:27:30.163 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.160+0000 7f1b82ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:30.164 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.160+0000 7f1b82ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:27:30.164 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.160+0000 7f1b82ffd640 1 -- 172.21.15.155:0/1667209371 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b8c1d83d0 con 0x7f1b8c13f710 2024-03-25T18:27:30.164 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.160+0000 7f1b82ffd640 10 monclient: _finish_auth 0 2024-03-25T18:27:30.164 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.160+0000 7f1b82ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:27:30.164 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.160+0000 7f1b82ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:30.164 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.160+0000 7f1b94720640 5 monclient: authenticate success, global_id 4339 2024-03-25T18:27:30.164 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.160+0000 7f1b94720640 10 monclient: _renew_subs 2024-03-25T18:27:30.164 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.161+0000 7f1b94720640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:27:30.164 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.161+0000 7f1b94720640 1 -- 172.21.15.155:0/1667209371 --> v1:172.21.15.167:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b8c1d41e0 con 0x7f1b8c13f710 2024-03-25T18:27:30.164 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.161+0000 7f1b94720640 10 client.4339.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:30.164 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.161+0000 7f1b94720640 10 monclient: _renew_subs 2024-03-25T18:27:30.165 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.161+0000 7f1b94720640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:27:30.165 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.161+0000 7f1b94720640 1 -- 172.21.15.155:0/1667209371 --> v1:172.21.15.167:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b8c1d5110 con 0x7f1b8c13f710 2024-03-25T18:27:30.165 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.161+0000 7f1b94720640 1 librados: init done 2024-03-25T18:27:30.165 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.162+0000 7f1b94720640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:30.165 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.162+0000 7f1b94720640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:30.165 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.162+0000 7f1b94720640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:27:30.165 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.162+0000 7f1b94720640 1 -- 172.21.15.155:0/1667209371 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b8c1d7930 con 0x7f1b8c13f710 2024-03-25T18:27:30.434 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.433+0000 7f1b82ffd640 1 -- 172.21.15.155:0/1667209371 <== mon.1 v1:172.21.15.167:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b7c0011f0 con 0x7f1b8c13f710 2024-03-25T18:27:30.434 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.433+0000 7f1b82ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:30.434 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.433+0000 7f1b82ffd640 1 -- 172.21.15.155:0/1667209371 <== mon.1 v1:172.21.15.167:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f1b7c004cb0 con 0x7f1b8c13f710 2024-03-25T18:27:30.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.433+0000 7f1b82ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:30.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.433+0000 7f1b82ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:30.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.433+0000 7f1b82ffd640 10 monclient: dump: 2024-03-25T18:27:30.435 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:27:30.435 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:27:30.435 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:30.435 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:30.435 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:30.435 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:27:30.435 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:27:30.435 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:27:30.435 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:27:30.435 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:27:30.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.433+0000 7f1b90cb9640 4 set_mon_vals no callback set 2024-03-25T18:27:30.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.434+0000 7f1b82ffd640 1 -- 172.21.15.155:0/1667209371 <== mon.1 v1:172.21.15.167:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7f1b7c01ad00 con 0x7f1b8c13f710 2024-03-25T18:27:30.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.435+0000 7f1b924bc640 10 monclient: get_auth_request con 0x7f1b7006dd60 auth_method 0 2024-03-25T18:27:30.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.435+0000 7f1b82ffd640 10 client.4339.objecter ms_handle_connect 0x7f1b7006dd60 2024-03-25T18:27:30.443 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.442+0000 7f1b82ffd640 1 -- 172.21.15.155:0/1667209371 <== mon.1 v1:172.21.15.167:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7f1b7c0549d0 con 0x7f1b8c13f710 2024-03-25T18:27:30.443 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.442+0000 7f1b82ffd640 10 client.4339.objecter ms_dispatch 0x7f1b8c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:30.443 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.442+0000 7f1b82ffd640 3 client.4339.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:30.443 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.442+0000 7f1b82ffd640 3 client.4339.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:30.443 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.442+0000 7f1b82ffd640 20 client.4339.objecter dump_active .. 0 homeless 2024-03-25T18:27:30.444 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.443+0000 7f1b82ffd640 1 -- 172.21.15.155:0/1667209371 <== mon.1 v1:172.21.15.167:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1b7c050310 con 0x7f1b8c13f710 2024-03-25T18:27:30.444 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.443+0000 7f1b82ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:30.444 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.443+0000 7f1b82ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:30.702 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.701+0000 7f1b94720640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:30.702 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.701+0000 7f1b94720640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:30.702 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.701+0000 7f1b94720640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:27:30.702 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.701+0000 7f1b94720640 1 -- 172.21.15.155:0/1667209371 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1b8c14f320 con 0x7f1b8c13f710 2024-03-25T18:27:30.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.718+0000 7f1b82ffd640 1 -- 172.21.15.155:0/1667209371 <== mon.1 v1:172.21.15.167:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7f1b7c052410 con 0x7f1b8c13f710 2024-03-25T18:27:30.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.719+0000 7f1b82ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:30.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.719+0000 7f1b82ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:30.720 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:27:30.720 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:30.723 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.721+0000 7f1b94720640 10 librados: watch_flush enter 2024-03-25T18:27:30.723 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.721+0000 7f1b94720640 10 librados: watch_flush exit 2024-03-25T18:27:30.723 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.721+0000 7f1b94720640 20 client.4339.objecter shutdown clearing up homeless session... 2024-03-25T18:27:30.723 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.721+0000 7f1b94720640 10 client.4339.objecter successfully canceled tick 2024-03-25T18:27:30.723 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.721+0000 7f1b94720640 1 -- 172.21.15.155:0/1667209371 >> v1:172.21.15.167:6800/2919070503 conn(0x7f1b7006dd60 legacy=0x7f1b70070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:30.723 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.721+0000 7f1b94720640 10 monclient: shutdown 2024-03-25T18:27:30.723 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.721+0000 7f1b94720640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:30.723 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.721+0000 7f1b94720640 1 -- 172.21.15.155:0/1667209371 >> v1:172.21.15.167:6789/0 conn(0x7f1b8c13f710 legacy=0x7f1b8c1d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:30.723 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.721+0000 7f1b94720640 1 -- 172.21.15.155:0/1667209371 shutdown_connections 2024-03-25T18:27:30.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.721+0000 7f1b94720640 1 -- 172.21.15.155:0/1667209371 >> 172.21.15.155:0/1667209371 conn(0x7f1b8c0b7ea0 msgr2=0x7f1b8c13ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:30.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.722+0000 7f1b94720640 1 -- 172.21.15.155:0/1667209371 shutdown_connections 2024-03-25T18:27:30.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.722+0000 7f1b94720640 1 -- 172.21.15.155:0/1667209371 wait complete. 2024-03-25T18:27:30.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:30.722+0000 7f1b94720640 1 librados: shutdown 2024-03-25T18:27:30.735 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:37.543 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:37.658 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.657+0000 7f6c3afe6640 10 monclient: get_monmap_and_config 2024-03-25T18:27:37.658 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.657+0000 7f6c3afe6640 10 monclient: build_initial_monmap 2024-03-25T18:27:37.658 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.657+0000 7f6c3afe6640 10 monclient: monmap: 2024-03-25T18:27:37.658 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:27:37.659 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:27:37.659 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:27:37.658429+0000 2024-03-25T18:27:37.659 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:27:37.658429+0000 2024-03-25T18:27:37.659 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:37.659 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:27:37.659 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:27:37.659 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:27:37.659 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:27:37.659 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:27:37.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.658+0000 7f6c3afe6640 1 Processor -- start 2024-03-25T18:27:37.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.658+0000 7f6c3afe6640 1 -- start start 2024-03-25T18:27:37.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.658+0000 7f6c3afe6640 10 monclient: init 2024-03-25T18:27:37.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.659+0000 7f6c3afe6640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:37.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.659+0000 7f6c3afe6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:27:37.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.659+0000 7f6c3afe6640 10 monclient(hunting): picked mon.noname-b con 0x7f6c340a98f0 addr v1:172.21.15.155:6790/0 2024-03-25T18:27:37.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.659+0000 7f6c3afe6640 10 monclient(hunting): picked mon.noname-a con 0x7f6c3413f710 addr v1:172.21.15.155:6789/0 2024-03-25T18:27:37.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.659+0000 7f6c3afe6640 10 monclient(hunting): picked mon.noname-c con 0x7f6c341430c0 addr v1:172.21.15.167:6789/0 2024-03-25T18:27:37.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.659+0000 7f6c3afe6640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c34142520 con 0x7f6c3413f710 2024-03-25T18:27:37.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.659+0000 7f6c3afe6640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c340b28a0 con 0x7f6c341430c0 2024-03-25T18:27:37.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.659+0000 7f6c3afe6640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c340b2a30 con 0x7f6c340a98f0 2024-03-25T18:27:37.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.659+0000 7f6c3afe6640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:37.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.659+0000 7f6c3afe6640 10 monclient(hunting): authenticate will time out at 7703.969012s 2024-03-25T18:27:37.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.662+0000 7f6c33fff640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f6c3413f710 0x7f6c341429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:60430/0 (socket says 172.21.15.155:60430) 2024-03-25T18:27:37.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.662+0000 7f6c33fff640 1 -- 172.21.15.155:0/995784349 learned_addr learned my addr 172.21.15.155:0/995784349 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:27:37.670 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.669+0000 7f6c337fe640 1 -- 172.21.15.155:0/995784349 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1485898062 0 0) 0x7f6c34142520 con 0x7f6c3413f710 2024-03-25T18:27:37.670 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.669+0000 7f6c337fe640 10 monclient(hunting): _negotiate 2024-03-25T18:27:37.670 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.669+0000 7f6c337fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:37.670 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.669+0000 7f6c337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:37.670 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.669+0000 7f6c337fe640 10 monclient(hunting): my global_id is 4347 2024-03-25T18:27:37.670 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.669+0000 7f6c337fe640 1 -- 172.21.15.155:0/995784349 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c24003520 con 0x7f6c3413f710 2024-03-25T18:27:37.684 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.683+0000 7f6c337fe640 1 -- 172.21.15.155:0/995784349 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2279077056 0 0) 0x7f6c24003520 con 0x7f6c3413f710 2024-03-25T18:27:37.684 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.683+0000 7f6c337fe640 1 -- 172.21.15.155:0/995784349 >> v1:172.21.15.155:6790/0 conn(0x7f6c340a98f0 legacy=0x7f6c340a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:37.684 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.683+0000 7f6c337fe640 1 -- 172.21.15.155:0/995784349 >> v1:172.21.15.167:6789/0 conn(0x7f6c341430c0 legacy=0x7f6c3414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:37.684 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.683+0000 7f6c337fe640 10 monclient: _finish_hunting 0 2024-03-25T18:27:37.684 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.683+0000 7f6c337fe640 1 monclient: found mon.noname-a 2024-03-25T18:27:37.684 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.683+0000 7f6c337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:37.685 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.683+0000 7f6c337fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-25T18:27:37.685 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.683+0000 7f6c337fe640 1 -- 172.21.15.155:0/995784349 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c340b44c0 con 0x7f6c3413f710 2024-03-25T18:27:37.685 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.683+0000 7f6c337fe640 10 monclient: _finish_auth 0 2024-03-25T18:27:37.685 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.683+0000 7f6c337fe640 10 monclient: _check_auth_tickets 2024-03-25T18:27:37.685 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.683+0000 7f6c337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:37.685 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.683+0000 7f6c337fe640 1 -- 172.21.15.155:0/995784349 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f6c20001690 con 0x7f6c3413f710 2024-03-25T18:27:37.685 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.683+0000 7f6c337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:37.685 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.683+0000 7f6c337fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:27:37.685 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.683+0000 7f6c337fe640 10 monclient: dump: 2024-03-25T18:27:37.685 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:27:37.685 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:27:37.685 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:37.685 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:37.685 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:37.686 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:27:37.686 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:27:37.686 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:27:37.686 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:27:37.686 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:27:37.686 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.683+0000 7f6c3afe6640 5 monclient: authenticate success, global_id 4347 2024-03-25T18:27:37.686 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.683+0000 7f6c3afe6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:37.690 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.689+0000 7f6c337fe640 1 -- 172.21.15.155:0/995784349 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6c200017f0 con 0x7f6c3413f710 2024-03-25T18:27:37.690 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.689+0000 7f6c337fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:37.690 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.689+0000 7f6c337fe640 1 -- 172.21.15.155:0/995784349 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f6c20001c00 con 0x7f6c3413f710 2024-03-25T18:27:37.690 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.689+0000 7f6c3afe6640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:37.690 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.689+0000 7f6c3afe6640 4 set_mon_vals no callback set 2024-03-25T18:27:37.690 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.689+0000 7f6c337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:37.691 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.689+0000 7f6c337fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:37.691 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.689+0000 7f6c337fe640 10 monclient: dump: 2024-03-25T18:27:37.691 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:27:37.691 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:27:37.691 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:37.691 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:37.691 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:37.691 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:27:37.691 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:27:37.691 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:27:37.691 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:27:37.691 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:27:37.693 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.690+0000 7f6c3afe6640 10 monclient: shutdown 2024-03-25T18:27:37.693 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.690+0000 7f6c3afe6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:37.693 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.690+0000 7f6c3afe6640 1 -- 172.21.15.155:0/995784349 >> v1:172.21.15.155:6789/0 conn(0x7f6c3413f710 legacy=0x7f6c341429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:37.693 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.690+0000 7f6c3afe6640 1 -- 172.21.15.155:0/995784349 shutdown_connections 2024-03-25T18:27:37.693 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.690+0000 7f6c3afe6640 1 -- 172.21.15.155:0/995784349 >> 172.21.15.155:0/995784349 conn(0x7f6c340b7ea0 msgr2=0x7f6c340b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:37.693 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.690+0000 7f6c3afe6640 1 -- 172.21.15.155:0/995784349 shutdown_connections 2024-03-25T18:27:37.693 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.690+0000 7f6c3afe6640 1 -- 172.21.15.155:0/995784349 wait complete. 2024-03-25T18:27:37.693 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.691+0000 7f6c3afe6640 10 monclient: build_initial_monmap 2024-03-25T18:27:37.694 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.691+0000 7f6c3afe6640 10 monclient: monmap: 2024-03-25T18:27:37.694 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:27:37.694 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:37.694 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:27:37.694 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:27:37.694 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:37.694 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:27:37.694 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:27:37.694 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:27:37.694 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:27:37.694 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:27:37.694 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.691+0000 7f6c3afe6640 1 librados: starting msgr at 2024-03-25T18:27:37.694 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.691+0000 7f6c3afe6640 1 librados: starting objecter 2024-03-25T18:27:37.694 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.691+0000 7f6c3afe6640 1 Processor -- start 2024-03-25T18:27:37.695 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.691+0000 7f6c3afe6640 1 -- start start 2024-03-25T18:27:37.695 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.691+0000 7f6c3afe6640 1 librados: setting wanted keys 2024-03-25T18:27:37.695 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.691+0000 7f6c3afe6640 1 librados: calling monclient init 2024-03-25T18:27:37.695 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.691+0000 7f6c3afe6640 10 monclient: init 2024-03-25T18:27:37.696 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.692+0000 7f6c3afe6640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:37.696 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.692+0000 7f6c3afe6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:27:37.696 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.692+0000 7f6c3afe6640 10 monclient(hunting): picked mon.noname-a con 0x7f6c340a98f0 addr v1:172.21.15.155:6789/0 2024-03-25T18:27:37.696 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.692+0000 7f6c3afe6640 10 monclient(hunting): picked mon.noname-c con 0x7f6c3413f710 addr v1:172.21.15.155:6790/0 2024-03-25T18:27:37.696 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.693+0000 7f6c3afe6640 10 monclient(hunting): picked mon.noname-b con 0x7f6c341430c0 addr v1:172.21.15.167:6789/0 2024-03-25T18:27:37.696 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.693+0000 7f6c3afe6640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c34139a80 con 0x7f6c340a98f0 2024-03-25T18:27:37.696 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.693+0000 7f6c3afe6640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c34137450 con 0x7f6c341430c0 2024-03-25T18:27:37.696 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.693+0000 7f6c3afe6640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c341d7ba0 con 0x7f6c3413f710 2024-03-25T18:27:37.696 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.693+0000 7f6c3afe6640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:37.696 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.693+0000 7f6c3afe6640 10 monclient(hunting): authenticate will time out at 7704.002913s 2024-03-25T18:27:37.697 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.696+0000 7f6c33fff640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f6c3413f710 0x7f6c341d7490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:45472/0 (socket says 172.21.15.155:45472) 2024-03-25T18:27:37.697 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.696+0000 7f6c33fff640 1 -- 172.21.15.155:0/610834340 learned_addr learned my addr 172.21.15.155:0/610834340 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:27:37.698 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.697+0000 7f6c317fa640 10 client.?.objecter ms_handle_connect 0x7f6c3413f710 2024-03-25T18:27:37.698 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.697+0000 7f6c317fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:37.705 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.704+0000 7f6c317fa640 1 -- 172.21.15.155:0/610834340 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1061286292 0 0) 0x7f6c341d7ba0 con 0x7f6c3413f710 2024-03-25T18:27:37.705 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.704+0000 7f6c317fa640 10 monclient(hunting): _negotiate 2024-03-25T18:27:37.705 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.704+0000 7f6c317fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:37.705 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.704+0000 7f6c317fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:37.705 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.704+0000 7f6c317fa640 10 monclient(hunting): my global_id is 4358 2024-03-25T18:27:37.705 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.704+0000 7f6c317fa640 1 -- 172.21.15.155:0/610834340 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c18004eb0 con 0x7f6c3413f710 2024-03-25T18:27:37.712 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.712+0000 7f6c317fa640 10 client.?.objecter ms_handle_connect 0x7f6c341430c0 2024-03-25T18:27:37.712 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.712+0000 7f6c317fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:37.713 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.712+0000 7f6c317fa640 10 client.?.objecter ms_handle_connect 0x7f6c340a98f0 2024-03-25T18:27:37.713 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.712+0000 7f6c317fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:37.727 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.726+0000 7f6c317fa640 1 -- 172.21.15.155:0/610834340 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2532885024 0 0) 0x7f6c340b2a30 con 0x7f6c340a98f0 2024-03-25T18:27:37.729 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.726+0000 7f6c317fa640 10 monclient(hunting): _negotiate 2024-03-25T18:27:37.729 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.726+0000 7f6c317fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:37.729 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.726+0000 7f6c317fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:37.729 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.726+0000 7f6c317fa640 10 monclient(hunting): my global_id is 4350 2024-03-25T18:27:37.729 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.726+0000 7f6c317fa640 1 -- 172.21.15.155:0/610834340 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c341d7ba0 con 0x7f6c340a98f0 2024-03-25T18:27:37.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.730+0000 7f6c317fa640 1 -- 172.21.15.155:0/610834340 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 115358765 0 0) 0x7f6c34137450 con 0x7f6c341430c0 2024-03-25T18:27:37.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.730+0000 7f6c317fa640 10 monclient(hunting): _negotiate 2024-03-25T18:27:37.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.730+0000 7f6c317fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:37.732 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.730+0000 7f6c317fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:37.732 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.730+0000 7f6c317fa640 10 monclient(hunting): my global_id is 4345 2024-03-25T18:27:37.732 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.730+0000 7f6c317fa640 1 -- 172.21.15.155:0/610834340 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c340b2a30 con 0x7f6c341430c0 2024-03-25T18:27:37.734 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.733+0000 7f6c317fa640 1 -- 172.21.15.155:0/610834340 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1838627719 0 0) 0x7f6c18004eb0 con 0x7f6c3413f710 2024-03-25T18:27:37.734 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.733+0000 7f6c317fa640 1 -- 172.21.15.155:0/610834340 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6c34137450 con 0x7f6c3413f710 2024-03-25T18:27:37.734 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.733+0000 7f6c317fa640 1 -- 172.21.15.155:0/610834340 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f6c20001bc0 con 0x7f6c3413f710 2024-03-25T18:27:37.734 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.733+0000 7f6c317fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:37.734 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.733+0000 7f6c317fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:27:37.734 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.733+0000 7f6c317fa640 10 monclient(hunting): dump: 2024-03-25T18:27:37.735 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:27:37.735 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:27:37.735 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:37.735 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:37.735 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:37.735 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:27:37.735 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:27:37.735 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:27:37.735 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:27:37.735 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:27:37.735 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.733+0000 7f6c317fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:37.748 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.746+0000 7f6c317fa640 1 -- 172.21.15.155:0/610834340 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 359455449 0 0) 0x7f6c340b2a30 con 0x7f6c341430c0 2024-03-25T18:27:37.748 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.746+0000 7f6c317fa640 1 -- 172.21.15.155:0/610834340 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6c18004eb0 con 0x7f6c341430c0 2024-03-25T18:27:37.748 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.746+0000 7f6c317fa640 1 -- 172.21.15.155:0/610834340 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f6c2c001a90 con 0x7f6c341430c0 2024-03-25T18:27:37.749 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.746+0000 7f6c317fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:37.749 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.746+0000 7f6c317fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:37.749 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.746+0000 7f6c317fa640 10 monclient(hunting): dump: 2024-03-25T18:27:37.749 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:27:37.749 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:27:37.749 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:37.749 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:37.749 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:37.749 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:27:37.749 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:27:37.749 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:27:37.749 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:27:37.749 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:27:37.749 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.747+0000 7f6c317fa640 1 -- 172.21.15.155:0/610834340 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4027510330 0 0) 0x7f6c34137450 con 0x7f6c3413f710 2024-03-25T18:27:37.749 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.747+0000 7f6c317fa640 1 -- 172.21.15.155:0/610834340 >> v1:172.21.15.167:6789/0 conn(0x7f6c341430c0 legacy=0x7f6c34132f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:37.750 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.747+0000 7f6c317fa640 1 -- 172.21.15.155:0/610834340 >> v1:172.21.15.155:6789/0 conn(0x7f6c340a98f0 legacy=0x7f6c34139370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:37.750 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.747+0000 7f6c317fa640 10 monclient: _finish_hunting 0 2024-03-25T18:27:37.750 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.747+0000 7f6c317fa640 1 monclient: found mon.c 2024-03-25T18:27:37.750 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.747+0000 7f6c317fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:37.750 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.747+0000 7f6c317fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:27:37.750 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.747+0000 7f6c317fa640 1 -- 172.21.15.155:0/610834340 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c341368f0 con 0x7f6c3413f710 2024-03-25T18:27:37.750 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.747+0000 7f6c317fa640 10 monclient: _finish_auth 0 2024-03-25T18:27:37.750 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.747+0000 7f6c317fa640 10 monclient: _check_auth_tickets 2024-03-25T18:27:37.750 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.747+0000 7f6c317fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:37.750 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.747+0000 7f6c3afe6640 5 monclient: authenticate success, global_id 4358 2024-03-25T18:27:37.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.747+0000 7f6c3afe6640 10 monclient: _renew_subs 2024-03-25T18:27:37.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.747+0000 7f6c3afe6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:27:37.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.747+0000 7f6c3afe6640 1 -- 172.21.15.155:0/610834340 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c34136b40 con 0x7f6c3413f710 2024-03-25T18:27:37.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.747+0000 7f6c3afe6640 10 client.4358.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:37.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.747+0000 7f6c3afe6640 10 monclient: _renew_subs 2024-03-25T18:27:37.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.747+0000 7f6c3afe6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:27:37.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.747+0000 7f6c3afe6640 1 -- 172.21.15.155:0/610834340 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c34137bb0 con 0x7f6c3413f710 2024-03-25T18:27:37.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.747+0000 7f6c3afe6640 1 librados: init done 2024-03-25T18:27:37.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.748+0000 7f6c3afe6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:37.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.748+0000 7f6c3afe6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:37.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.748+0000 7f6c3afe6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:27:37.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:37.748+0000 7f6c3afe6640 1 -- 172.21.15.155:0/610834340 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c340627b0 con 0x7f6c3413f710 2024-03-25T18:27:38.055 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:38.054+0000 7f6c317fa640 1 -- 172.21.15.155:0/610834340 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6c20004a90 con 0x7f6c3413f710 2024-03-25T18:27:38.055 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:38.054+0000 7f6c317fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:38.055 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:38.054+0000 7f6c317fa640 1 -- 172.21.15.155:0/610834340 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f6c20004c00 con 0x7f6c3413f710 2024-03-25T18:27:38.055 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:38.054+0000 7f6c317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:38.055 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:38.054+0000 7f6c317fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:38.055 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:38.054+0000 7f6c317fa640 10 monclient: dump: 2024-03-25T18:27:38.055 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:27:38.055 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:27:38.055 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:38.056 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:38.056 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:38.056 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:27:38.056 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:27:38.056 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:27:38.056 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:27:38.056 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:27:38.056 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:38.054+0000 7f6c32ffd640 4 set_mon_vals no callback set 2024-03-25T18:27:38.057 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:38.055+0000 7f6c317fa640 1 -- 172.21.15.155:0/610834340 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7f6c2001abd0 con 0x7f6c3413f710 2024-03-25T18:27:38.057 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:38.055+0000 7f6c38d82640 10 monclient: get_auth_request con 0x7f6c1806dfb0 auth_method 0 2024-03-25T18:27:38.057 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:38.056+0000 7f6c317fa640 10 client.4358.objecter ms_handle_connect 0x7f6c1806dfb0 2024-03-25T18:27:38.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:38.060+0000 7f6c317fa640 1 -- 172.21.15.155:0/610834340 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7f6c20054730 con 0x7f6c3413f710 2024-03-25T18:27:38.062 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:38.060+0000 7f6c317fa640 10 client.4358.objecter ms_dispatch 0x7f6c34000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:38.062 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:38.060+0000 7f6c317fa640 3 client.4358.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:38.062 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:38.060+0000 7f6c317fa640 3 client.4358.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:38.062 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:38.060+0000 7f6c317fa640 20 client.4358.objecter dump_active .. 0 homeless 2024-03-25T18:27:38.062 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:38.060+0000 7f6c317fa640 1 -- 172.21.15.155:0/610834340 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6c20050240 con 0x7f6c3413f710 2024-03-25T18:27:38.062 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:38.060+0000 7f6c317fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:38.062 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:38.060+0000 7f6c317fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:38.302 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:38.301+0000 7f6c3afe6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:38.302 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:38.301+0000 7f6c3afe6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:38.302 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:38.301+0000 7f6c3afe6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:27:38.302 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:38.301+0000 7f6c3afe6640 1 -- 172.21.15.155:0/610834340 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6c34074cc0 con 0x7f6c3413f710 2024-03-25T18:27:38.347 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:38.346+0000 7f6c317fa640 1 -- 172.21.15.155:0/610834340 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7f6c20052170 con 0x7f6c3413f710 2024-03-25T18:27:38.347 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:38.346+0000 7f6c317fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:38.347 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:38.346+0000 7f6c317fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:38.347 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:27:38.347 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:38.350 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:38.348+0000 7f6c3afe6640 10 librados: watch_flush enter 2024-03-25T18:27:38.350 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:38.348+0000 7f6c3afe6640 10 librados: watch_flush exit 2024-03-25T18:27:38.350 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:38.348+0000 7f6c3afe6640 20 client.4358.objecter shutdown clearing up homeless session... 2024-03-25T18:27:38.350 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:38.348+0000 7f6c3afe6640 10 client.4358.objecter successfully canceled tick 2024-03-25T18:27:38.350 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:38.348+0000 7f6c3afe6640 1 -- 172.21.15.155:0/610834340 >> v1:172.21.15.167:6800/2919070503 conn(0x7f6c1806dfb0 legacy=0x7f6c18070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:38.350 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:38.349+0000 7f6c3afe6640 10 monclient: shutdown 2024-03-25T18:27:38.350 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:38.349+0000 7f6c3afe6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:38.350 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:38.349+0000 7f6c3afe6640 1 -- 172.21.15.155:0/610834340 >> v1:172.21.15.155:6790/0 conn(0x7f6c3413f710 legacy=0x7f6c341d7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:38.350 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:38.349+0000 7f6c3afe6640 1 -- 172.21.15.155:0/610834340 shutdown_connections 2024-03-25T18:27:38.350 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:38.349+0000 7f6c3afe6640 1 -- 172.21.15.155:0/610834340 >> 172.21.15.155:0/610834340 conn(0x7f6c340b7ea0 msgr2=0x7f6c34074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:38.351 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:38.349+0000 7f6c3afe6640 1 -- 172.21.15.155:0/610834340 shutdown_connections 2024-03-25T18:27:38.351 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:38.349+0000 7f6c3afe6640 1 -- 172.21.15.155:0/610834340 wait complete. 2024-03-25T18:27:38.351 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:38.349+0000 7f6c3afe6640 1 librados: shutdown 2024-03-25T18:27:38.360 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:40.857 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:03:14.461 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:40.864 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:03:14.469 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:40.895 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:03:14.499 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:40.920 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:03:14.525 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:40.924 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:03:14.502 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:40.929 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:03:14.508 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:40.932 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:03:14.537 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:40.958 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:03:14.536 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:40.965 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:03:14.570 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:45.167 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:45.270 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.269+0000 7f5dbe779640 10 monclient: get_monmap_and_config 2024-03-25T18:27:45.271 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.269+0000 7f5dbe779640 10 monclient: build_initial_monmap 2024-03-25T18:27:45.271 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.269+0000 7f5dbe779640 10 monclient: monmap: 2024-03-25T18:27:45.271 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:27:45.271 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:27:45.271 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:27:45.270698+0000 2024-03-25T18:27:45.271 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:27:45.270698+0000 2024-03-25T18:27:45.271 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:45.271 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:27:45.271 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:27:45.271 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:27:45.271 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:27:45.271 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:27:45.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.270+0000 7f5dbe779640 1 Processor -- start 2024-03-25T18:27:45.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.270+0000 7f5dbe779640 1 -- start start 2024-03-25T18:27:45.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.270+0000 7f5dbe779640 10 monclient: init 2024-03-25T18:27:45.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.271+0000 7f5dbe779640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:45.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.271+0000 7f5dbe779640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:27:45.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.271+0000 7f5dbe779640 10 monclient(hunting): picked mon.noname-c con 0x7f5db80b4d60 addr v1:172.21.15.167:6789/0 2024-03-25T18:27:45.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.271+0000 7f5dbe779640 10 monclient(hunting): picked mon.noname-a con 0x7f5db80b1730 addr v1:172.21.15.155:6789/0 2024-03-25T18:27:45.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.271+0000 7f5dbe779640 10 monclient(hunting): picked mon.noname-b con 0x7f5db814be90 addr v1:172.21.15.155:6790/0 2024-03-25T18:27:45.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.271+0000 7f5dbe779640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5db80af940 con 0x7f5db80b1730 2024-03-25T18:27:45.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.271+0000 7f5dbe779640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5db80b0a70 con 0x7f5db80b4d60 2024-03-25T18:27:45.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.271+0000 7f5dbe779640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5db80b0d50 con 0x7f5db814be90 2024-03-25T18:27:45.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.271+0000 7f5dbe779640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:45.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.271+0000 7f5dbe779640 10 monclient(hunting): authenticate will time out at 7711.581253s 2024-03-25T18:27:45.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.272+0000 7f5dbcd16640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f5db814be90 0x7f5db814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:50188/0 (socket says 172.21.15.155:50188) 2024-03-25T18:27:45.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.272+0000 7f5dbcd16640 1 -- 172.21.15.155:0/1765785769 learned_addr learned my addr 172.21.15.155:0/1765785769 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:27:45.314 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.312+0000 7f5db6ffd640 1 -- 172.21.15.155:0/1765785769 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3481917222 0 0) 0x7f5db80af940 con 0x7f5db80b1730 2024-03-25T18:27:45.314 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.312+0000 7f5db6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:45.314 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.312+0000 7f5db6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:45.314 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.312+0000 7f5db6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:45.314 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.312+0000 7f5db6ffd640 10 monclient(hunting): my global_id is 4353 2024-03-25T18:27:45.314 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.312+0000 7f5db6ffd640 1 -- 172.21.15.155:0/1765785769 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5da0003520 con 0x7f5db80b1730 2024-03-25T18:27:45.321 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.319+0000 7f5db6ffd640 1 -- 172.21.15.155:0/1765785769 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4075588920 0 0) 0x7f5da0003520 con 0x7f5db80b1730 2024-03-25T18:27:45.321 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.319+0000 7f5db6ffd640 1 -- 172.21.15.155:0/1765785769 >> v1:172.21.15.155:6790/0 conn(0x7f5db814be90 legacy=0x7f5db814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:45.321 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.319+0000 7f5db6ffd640 1 -- 172.21.15.155:0/1765785769 >> v1:172.21.15.167:6789/0 conn(0x7f5db80b4d60 legacy=0x7f5db813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:45.321 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.319+0000 7f5db6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:27:45.321 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.319+0000 7f5db6ffd640 1 monclient: found mon.noname-a 2024-03-25T18:27:45.321 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.319+0000 7f5db6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:45.322 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.319+0000 7f5db6ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-25T18:27:45.322 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.319+0000 7f5db6ffd640 1 -- 172.21.15.155:0/1765785769 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5db80b1030 con 0x7f5db80b1730 2024-03-25T18:27:45.322 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.319+0000 7f5db6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:27:45.322 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.319+0000 7f5db6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:27:45.322 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.319+0000 7f5db6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:45.322 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.319+0000 7f5db6ffd640 1 -- 172.21.15.155:0/1765785769 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f5dac001690 con 0x7f5db80b1730 2024-03-25T18:27:45.322 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.319+0000 7f5db6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:45.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.319+0000 7f5db6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:27:45.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.319+0000 7f5db6ffd640 10 monclient: dump: 2024-03-25T18:27:45.323 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:27:45.323 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:27:45.323 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:45.323 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:45.323 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:45.324 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:27:45.324 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:27:45.324 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:27:45.324 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:27:45.325 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:27:45.325 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.319+0000 7f5dbe779640 5 monclient: authenticate success, global_id 4353 2024-03-25T18:27:45.325 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.319+0000 7f5dbe779640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:45.331 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.330+0000 7f5db6ffd640 1 -- 172.21.15.155:0/1765785769 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5dac0017f0 con 0x7f5db80b1730 2024-03-25T18:27:45.331 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.330+0000 7f5db6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:45.332 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.330+0000 7f5db6ffd640 1 -- 172.21.15.155:0/1765785769 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f5dac001c00 con 0x7f5db80b1730 2024-03-25T18:27:45.332 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.330+0000 7f5dbe779640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:45.332 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.330+0000 7f5dbe779640 4 set_mon_vals no callback set 2024-03-25T18:27:45.332 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.330+0000 7f5db6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:45.332 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.330+0000 7f5db6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:45.332 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.330+0000 7f5db6ffd640 10 monclient: dump: 2024-03-25T18:27:45.332 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:27:45.332 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:27:45.332 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:45.332 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:45.332 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:45.332 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:27:45.332 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:27:45.332 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:27:45.333 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:27:45.333 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:27:45.335 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.331+0000 7f5dbe779640 10 monclient: shutdown 2024-03-25T18:27:45.335 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.331+0000 7f5dbe779640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:45.335 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.331+0000 7f5dbe779640 1 -- 172.21.15.155:0/1765785769 >> v1:172.21.15.155:6789/0 conn(0x7f5db80b1730 legacy=0x7f5db80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:45.335 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.331+0000 7f5dbe779640 1 -- 172.21.15.155:0/1765785769 shutdown_connections 2024-03-25T18:27:45.335 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.331+0000 7f5dbe779640 1 -- 172.21.15.155:0/1765785769 >> 172.21.15.155:0/1765785769 conn(0x7f5db80617a0 msgr2=0x7f5db8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:45.335 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.331+0000 7f5dbe779640 1 -- 172.21.15.155:0/1765785769 shutdown_connections 2024-03-25T18:27:45.336 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.331+0000 7f5dbe779640 1 -- 172.21.15.155:0/1765785769 wait complete. 2024-03-25T18:27:45.336 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.332+0000 7f5dbe779640 10 monclient: build_initial_monmap 2024-03-25T18:27:45.336 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.332+0000 7f5dbe779640 10 monclient: monmap: 2024-03-25T18:27:45.336 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:27:45.336 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:45.336 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:27:45.336 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:27:45.336 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:45.336 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:27:45.336 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:27:45.336 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:27:45.336 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:27:45.336 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:27:45.336 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.332+0000 7f5dbe779640 1 librados: starting msgr at 2024-03-25T18:27:45.337 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.332+0000 7f5dbe779640 1 librados: starting objecter 2024-03-25T18:27:45.337 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.332+0000 7f5dbe779640 1 Processor -- start 2024-03-25T18:27:45.337 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.332+0000 7f5dbe779640 1 -- start start 2024-03-25T18:27:45.337 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.332+0000 7f5dbe779640 1 librados: setting wanted keys 2024-03-25T18:27:45.337 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.332+0000 7f5dbe779640 1 librados: calling monclient init 2024-03-25T18:27:45.337 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.332+0000 7f5dbe779640 10 monclient: init 2024-03-25T18:27:45.337 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.333+0000 7f5dbe779640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:45.337 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.333+0000 7f5dbe779640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:27:45.337 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.333+0000 7f5dbe779640 10 monclient(hunting): picked mon.noname-b con 0x7f5db80b1730 addr v1:172.21.15.167:6789/0 2024-03-25T18:27:45.337 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.333+0000 7f5dbe779640 10 monclient(hunting): picked mon.noname-a con 0x7f5db80b4d60 addr v1:172.21.15.155:6789/0 2024-03-25T18:27:45.337 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.333+0000 7f5dbe779640 10 monclient(hunting): picked mon.noname-c con 0x7f5db814be90 addr v1:172.21.15.155:6790/0 2024-03-25T18:27:45.337 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.333+0000 7f5dbe779640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5db8136e40 con 0x7f5db80b4d60 2024-03-25T18:27:45.337 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.333+0000 7f5dbe779640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5db81307c0 con 0x7f5db80b1730 2024-03-25T18:27:45.337 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.333+0000 7f5dbe779640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5db8138610 con 0x7f5db814be90 2024-03-25T18:27:45.337 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.333+0000 7f5dbe779640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:45.338 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.333+0000 7f5dbe779640 10 monclient(hunting): authenticate will time out at 7711.643420s 2024-03-25T18:27:45.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.352+0000 7f5db77fe640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f5db80b4d60 0x7f5db812f9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:36906/0 (socket says 172.21.15.155:36906) 2024-03-25T18:27:45.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.352+0000 7f5db77fe640 1 -- 172.21.15.155:0/3616749927 learned_addr learned my addr 172.21.15.155:0/3616749927 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:27:45.357 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.357+0000 7f5db4ff9640 10 client.?.objecter ms_handle_connect 0x7f5db80b4d60 2024-03-25T18:27:45.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.357+0000 7f5db4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:45.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.360+0000 7f5db4ff9640 10 client.?.objecter ms_handle_connect 0x7f5db80b1730 2024-03-25T18:27:45.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.360+0000 7f5db4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:45.372 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.371+0000 7f5db4ff9640 1 -- 172.21.15.155:0/3616749927 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2868175100 0 0) 0x7f5db8136e40 con 0x7f5db80b4d60 2024-03-25T18:27:45.372 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.371+0000 7f5db4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:27:45.372 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.371+0000 7f5db4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:45.373 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.371+0000 7f5db4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:45.373 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.371+0000 7f5db4ff9640 10 monclient(hunting): my global_id is 4356 2024-03-25T18:27:45.373 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.372+0000 7f5db4ff9640 1 -- 172.21.15.155:0/3616749927 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5da4004eb0 con 0x7f5db80b4d60 2024-03-25T18:27:45.374 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.373+0000 7f5db4ff9640 10 client.?.objecter ms_handle_connect 0x7f5db814be90 2024-03-25T18:27:45.374 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.373+0000 7f5db4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:45.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.453+0000 7f5db4ff9640 1 -- 172.21.15.155:0/3616749927 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2209309104 0 0) 0x7f5db8138610 con 0x7f5db814be90 2024-03-25T18:27:45.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.453+0000 7f5db4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:27:45.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.453+0000 7f5db4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:45.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.453+0000 7f5db4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:45.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.453+0000 7f5db4ff9640 10 monclient(hunting): my global_id is 4364 2024-03-25T18:27:45.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.453+0000 7f5db4ff9640 1 -- 172.21.15.155:0/3616749927 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5db8136e40 con 0x7f5db814be90 2024-03-25T18:27:45.462 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.461+0000 7f5db4ff9640 1 -- 172.21.15.155:0/3616749927 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3945648930 0 0) 0x7f5da4004eb0 con 0x7f5db80b4d60 2024-03-25T18:27:45.462 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.461+0000 7f5db4ff9640 1 -- 172.21.15.155:0/3616749927 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5db8138610 con 0x7f5db80b4d60 2024-03-25T18:27:45.462 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.461+0000 7f5db4ff9640 1 -- 172.21.15.155:0/3616749927 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f5dac001690 con 0x7f5db80b4d60 2024-03-25T18:27:45.463 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.461+0000 7f5db4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:45.463 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.461+0000 7f5db4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:27:45.463 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.461+0000 7f5db4ff9640 10 monclient(hunting): dump: 2024-03-25T18:27:45.463 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:27:45.463 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:27:45.463 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:45.463 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:45.463 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:45.463 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:27:45.463 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:27:45.463 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:27:45.463 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:27:45.463 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:27:45.463 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.461+0000 7f5db4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:45.464 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.463+0000 7f5db4ff9640 1 -- 172.21.15.155:0/3616749927 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1860852997 0 0) 0x7f5db81307c0 con 0x7f5db80b1730 2024-03-25T18:27:45.464 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.463+0000 7f5db4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:27:45.464 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.463+0000 7f5db4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:45.464 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.463+0000 7f5db4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:45.464 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.463+0000 7f5db4ff9640 10 monclient(hunting): my global_id is 4351 2024-03-25T18:27:45.464 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.463+0000 7f5db4ff9640 1 -- 172.21.15.155:0/3616749927 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5da4004eb0 con 0x7f5db80b1730 2024-03-25T18:27:45.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.484+0000 7f5db4ff9640 1 -- 172.21.15.155:0/3616749927 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3844033320 0 0) 0x7f5da4004eb0 con 0x7f5db80b1730 2024-03-25T18:27:45.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.484+0000 7f5db4ff9640 1 -- 172.21.15.155:0/3616749927 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5db81307c0 con 0x7f5db80b1730 2024-03-25T18:27:45.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.485+0000 7f5db4ff9640 1 -- 172.21.15.155:0/3616749927 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f5da8001860 con 0x7f5db80b1730 2024-03-25T18:27:45.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.485+0000 7f5db4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:45.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.485+0000 7f5db4ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:45.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.485+0000 7f5db4ff9640 10 monclient(hunting): dump: 2024-03-25T18:27:45.486 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:27:45.486 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:27:45.486 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:45.486 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:45.486 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:45.486 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:27:45.487 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:27:45.487 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:27:45.487 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:27:45.487 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:27:45.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.489+0000 7f5db4ff9640 1 -- 172.21.15.155:0/3616749927 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1952432646 0 0) 0x7f5db8138610 con 0x7f5db80b4d60 2024-03-25T18:27:45.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.489+0000 7f5db4ff9640 1 -- 172.21.15.155:0/3616749927 >> v1:172.21.15.155:6790/0 conn(0x7f5db814be90 legacy=0x7f5db81300b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:45.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.489+0000 7f5db4ff9640 1 -- 172.21.15.155:0/3616749927 >> v1:172.21.15.167:6789/0 conn(0x7f5db80b1730 legacy=0x7f5db81409d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:45.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.489+0000 7f5db4ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:27:45.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.489+0000 7f5db4ff9640 1 monclient: found mon.a 2024-03-25T18:27:45.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.489+0000 7f5db4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:45.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.489+0000 7f5db4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:27:45.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.489+0000 7f5db4ff9640 1 -- 172.21.15.155:0/3616749927 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5db81388f0 con 0x7f5db80b4d60 2024-03-25T18:27:45.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.489+0000 7f5db4ff9640 10 monclient: _finish_auth 0 2024-03-25T18:27:45.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.489+0000 7f5db4ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:27:45.492 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.489+0000 7f5db4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:45.492 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.489+0000 7f5dbe779640 5 monclient: authenticate success, global_id 4356 2024-03-25T18:27:45.492 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.489+0000 7f5dbe779640 10 monclient: _renew_subs 2024-03-25T18:27:45.492 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.489+0000 7f5dbe779640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:27:45.492 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.489+0000 7f5dbe779640 1 -- 172.21.15.155:0/3616749927 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5db81410e0 con 0x7f5db80b4d60 2024-03-25T18:27:45.492 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.490+0000 7f5dbe779640 10 client.4356.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:45.492 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.490+0000 7f5dbe779640 10 monclient: _renew_subs 2024-03-25T18:27:45.492 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.490+0000 7f5dbe779640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:27:45.492 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.490+0000 7f5dbe779640 1 -- 172.21.15.155:0/3616749927 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5db81381a0 con 0x7f5db80b4d60 2024-03-25T18:27:45.492 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.490+0000 7f5dbe779640 1 librados: init done 2024-03-25T18:27:45.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.491+0000 7f5dbe779640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:45.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.491+0000 7f5dbe779640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:45.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.491+0000 7f5dbe779640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:27:45.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.491+0000 7f5dbe779640 1 -- 172.21.15.155:0/3616749927 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5db81401f0 con 0x7f5db80b4d60 2024-03-25T18:27:45.839 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.838+0000 7f5db4ff9640 1 -- 172.21.15.155:0/3616749927 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5dac0038a0 con 0x7f5db80b4d60 2024-03-25T18:27:45.840 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.838+0000 7f5db4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:45.840 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.838+0000 7f5db4ff9640 1 -- 172.21.15.155:0/3616749927 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f5dac003cf0 con 0x7f5db80b4d60 2024-03-25T18:27:45.840 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.838+0000 7f5db4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:45.840 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.838+0000 7f5db4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:45.840 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.838+0000 7f5db4ff9640 10 monclient: dump: 2024-03-25T18:27:45.840 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:27:45.840 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:27:45.840 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:45.840 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:45.840 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:45.840 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:27:45.841 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:27:45.841 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:27:45.841 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:27:45.841 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:27:45.841 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.838+0000 7f5db67fc640 4 set_mon_vals no callback set 2024-03-25T18:27:45.841 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.839+0000 7f5db4ff9640 1 -- 172.21.15.155:0/3616749927 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7f5dac003f10 con 0x7f5db80b4d60 2024-03-25T18:27:45.842 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.840+0000 7f5db7fff640 10 monclient: get_auth_request con 0x7f5da406dfb0 auth_method 0 2024-03-25T18:27:45.842 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.840+0000 7f5db4ff9640 10 client.4356.objecter ms_handle_connect 0x7f5da406dfb0 2024-03-25T18:27:45.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.845+0000 7f5db4ff9640 1 -- 172.21.15.155:0/3616749927 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7f5dac0549a0 con 0x7f5db80b4d60 2024-03-25T18:27:45.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.845+0000 7f5db4ff9640 10 client.4356.objecter ms_dispatch 0x7f5db8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:45.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.845+0000 7f5db4ff9640 3 client.4356.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:45.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.845+0000 7f5db4ff9640 3 client.4356.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:45.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.845+0000 7f5db4ff9640 20 client.4356.objecter dump_active .. 0 homeless 2024-03-25T18:27:45.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.845+0000 7f5db4ff9640 1 -- 172.21.15.155:0/3616749927 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5dac0504b0 con 0x7f5db80b4d60 2024-03-25T18:27:45.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.845+0000 7f5db4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:45.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:45.845+0000 7f5db4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:46.101 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:46.100+0000 7f5dbe779640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:46.102 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:46.100+0000 7f5dbe779640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:46.102 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:46.100+0000 7f5dbe779640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:27:46.102 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:46.100+0000 7f5dbe779640 1 -- 172.21.15.155:0/3616749927 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5db80627b0 con 0x7f5db80b4d60 2024-03-25T18:27:46.146 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:46.145+0000 7f5db4ff9640 1 -- 172.21.15.155:0/3616749927 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7f5dac0523e0 con 0x7f5db80b4d60 2024-03-25T18:27:46.146 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:46.145+0000 7f5db4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:46.146 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:46.145+0000 7f5db4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:46.147 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:27:46.147 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:46.149 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:46.147+0000 7f5dbe779640 10 librados: watch_flush enter 2024-03-25T18:27:46.149 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:46.147+0000 7f5dbe779640 10 librados: watch_flush exit 2024-03-25T18:27:46.149 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:46.147+0000 7f5dbe779640 20 client.4356.objecter shutdown clearing up homeless session... 2024-03-25T18:27:46.149 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:46.147+0000 7f5dbe779640 10 client.4356.objecter successfully canceled tick 2024-03-25T18:27:46.149 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:46.147+0000 7f5dbe779640 1 -- 172.21.15.155:0/3616749927 >> v1:172.21.15.167:6800/2919070503 conn(0x7f5da406dfb0 legacy=0x7f5da4070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:46.149 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:46.147+0000 7f5dbe779640 10 monclient: shutdown 2024-03-25T18:27:46.149 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:46.147+0000 7f5dbe779640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:46.149 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:46.147+0000 7f5dbe779640 1 -- 172.21.15.155:0/3616749927 >> v1:172.21.15.155:6789/0 conn(0x7f5db80b4d60 legacy=0x7f5db812f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:46.149 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:46.148+0000 7f5dbe779640 1 -- 172.21.15.155:0/3616749927 shutdown_connections 2024-03-25T18:27:46.149 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:46.148+0000 7f5dbe779640 1 -- 172.21.15.155:0/3616749927 >> 172.21.15.155:0/3616749927 conn(0x7f5db80617a0 msgr2=0x7f5db80b25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:46.150 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:46.148+0000 7f5dbe779640 1 -- 172.21.15.155:0/3616749927 shutdown_connections 2024-03-25T18:27:46.150 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:46.148+0000 7f5dbe779640 1 -- 172.21.15.155:0/3616749927 wait complete. 2024-03-25T18:27:46.150 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:46.148+0000 7f5dbe779640 1 librados: shutdown 2024-03-25T18:27:46.160 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:52.968 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:53.084 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.083+0000 7feb93ec3640 10 monclient: get_monmap_and_config 2024-03-25T18:27:53.084 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.083+0000 7feb93ec3640 10 monclient: build_initial_monmap 2024-03-25T18:27:53.085 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.083+0000 7feb93ec3640 10 monclient: monmap: 2024-03-25T18:27:53.085 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:27:53.085 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:27:53.085 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:27:53.084750+0000 2024-03-25T18:27:53.085 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:27:53.084750+0000 2024-03-25T18:27:53.085 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:53.085 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:27:53.085 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:27:53.085 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:27:53.085 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:27:53.085 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:27:53.087 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.084+0000 7feb93ec3640 1 Processor -- start 2024-03-25T18:27:53.087 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.084+0000 7feb93ec3640 1 -- start start 2024-03-25T18:27:53.087 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.084+0000 7feb93ec3640 10 monclient: init 2024-03-25T18:27:53.087 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.085+0000 7feb93ec3640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:53.087 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.085+0000 7feb93ec3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:27:53.087 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.085+0000 7feb93ec3640 10 monclient(hunting): picked mon.noname-c con 0x7feb8c0b4d60 addr v1:172.21.15.167:6789/0 2024-03-25T18:27:53.087 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.085+0000 7feb93ec3640 10 monclient(hunting): picked mon.noname-a con 0x7feb8c0b1730 addr v1:172.21.15.155:6789/0 2024-03-25T18:27:53.088 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.085+0000 7feb93ec3640 10 monclient(hunting): picked mon.noname-b con 0x7feb8c14be90 addr v1:172.21.15.155:6790/0 2024-03-25T18:27:53.088 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.085+0000 7feb93ec3640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb8c0af940 con 0x7feb8c0b1730 2024-03-25T18:27:53.088 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.085+0000 7feb93ec3640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb8c0b0a70 con 0x7feb8c0b4d60 2024-03-25T18:27:53.088 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.085+0000 7feb93ec3640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb8c0b0d50 con 0x7feb8c14be90 2024-03-25T18:27:53.088 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.085+0000 7feb93ec3640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:53.088 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.085+0000 7feb93ec3640 10 monclient(hunting): authenticate will time out at 7719.395348s 2024-03-25T18:27:53.098 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.097+0000 7feb9145e640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7feb8c0b1730 0x7feb8c0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:50422/0 (socket says 172.21.15.155:50422) 2024-03-25T18:27:53.098 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.097+0000 7feb9145e640 1 -- 172.21.15.155:0/2736255529 learned_addr learned my addr 172.21.15.155:0/2736255529 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:27:53.109 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.108+0000 7feb90c5d640 1 -- 172.21.15.155:0/2736255529 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 769483441 0 0) 0x7feb8c0b0a70 con 0x7feb8c0b4d60 2024-03-25T18:27:53.109 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.108+0000 7feb90c5d640 10 monclient(hunting): _negotiate 2024-03-25T18:27:53.109 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.108+0000 7feb90c5d640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:53.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.108+0000 7feb90c5d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:53.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.108+0000 7feb90c5d640 10 monclient(hunting): my global_id is 4354 2024-03-25T18:27:53.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.108+0000 7feb90c5d640 1 -- 172.21.15.155:0/2736255529 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb78003520 con 0x7feb8c0b4d60 2024-03-25T18:27:53.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.114+0000 7feb90c5d640 1 -- 172.21.15.155:0/2736255529 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 873458555 0 0) 0x7feb8c0b0d50 con 0x7feb8c14be90 2024-03-25T18:27:53.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.114+0000 7feb90c5d640 10 monclient(hunting): _negotiate 2024-03-25T18:27:53.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.114+0000 7feb90c5d640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:53.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.114+0000 7feb90c5d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:53.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.114+0000 7feb90c5d640 10 monclient(hunting): my global_id is 4367 2024-03-25T18:27:53.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.114+0000 7feb90c5d640 1 -- 172.21.15.155:0/2736255529 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb8c0b0a70 con 0x7feb8c14be90 2024-03-25T18:27:53.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.119+0000 7feb90c5d640 1 -- 172.21.15.155:0/2736255529 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1061838962 0 0) 0x7feb8c0af940 con 0x7feb8c0b1730 2024-03-25T18:27:53.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.119+0000 7feb90c5d640 10 monclient(hunting): _negotiate 2024-03-25T18:27:53.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.119+0000 7feb90c5d640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:53.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.119+0000 7feb90c5d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:53.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.119+0000 7feb90c5d640 10 monclient(hunting): my global_id is 4359 2024-03-25T18:27:53.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.119+0000 7feb90c5d640 1 -- 172.21.15.155:0/2736255529 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb8c0b0d50 con 0x7feb8c0b1730 2024-03-25T18:27:53.195 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.194+0000 7feb90c5d640 1 -- 172.21.15.155:0/2736255529 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 585504118 0 0) 0x7feb8c0b0d50 con 0x7feb8c0b1730 2024-03-25T18:27:53.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.194+0000 7feb90c5d640 1 -- 172.21.15.155:0/2736255529 >> v1:172.21.15.155:6790/0 conn(0x7feb8c14be90 legacy=0x7feb8c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:53.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.194+0000 7feb90c5d640 1 -- 172.21.15.155:0/2736255529 >> v1:172.21.15.167:6789/0 conn(0x7feb8c0b4d60 legacy=0x7feb8c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:53.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.194+0000 7feb90c5d640 10 monclient: _finish_hunting 0 2024-03-25T18:27:53.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.194+0000 7feb90c5d640 1 monclient: found mon.noname-a 2024-03-25T18:27:53.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.194+0000 7feb90c5d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:53.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.194+0000 7feb90c5d640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-25T18:27:53.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.194+0000 7feb90c5d640 1 -- 172.21.15.155:0/2736255529 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb8c0b1030 con 0x7feb8c0b1730 2024-03-25T18:27:53.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.194+0000 7feb90c5d640 10 monclient: _finish_auth 0 2024-03-25T18:27:53.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.194+0000 7feb90c5d640 10 monclient: _check_auth_tickets 2024-03-25T18:27:53.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.194+0000 7feb90c5d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:53.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.194+0000 7feb90c5d640 1 -- 172.21.15.155:0/2736255529 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7feb7c0015b0 con 0x7feb8c0b1730 2024-03-25T18:27:53.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.194+0000 7feb90c5d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:53.197 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.194+0000 7feb90c5d640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:27:53.197 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.194+0000 7feb90c5d640 10 monclient: dump: 2024-03-25T18:27:53.197 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:27:53.197 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:27:53.197 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:53.197 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:53.197 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:53.197 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:27:53.197 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:27:53.197 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:27:53.197 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:27:53.197 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:27:53.197 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.194+0000 7feb93ec3640 5 monclient: authenticate success, global_id 4359 2024-03-25T18:27:53.197 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.194+0000 7feb93ec3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:53.207 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.206+0000 7feb90c5d640 1 -- 172.21.15.155:0/2736255529 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb7c001710 con 0x7feb8c0b1730 2024-03-25T18:27:53.207 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.206+0000 7feb90c5d640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:53.207 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.206+0000 7feb90c5d640 1 -- 172.21.15.155:0/2736255529 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7feb7c001b60 con 0x7feb8c0b1730 2024-03-25T18:27:53.207 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.206+0000 7feb93ec3640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:53.207 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.206+0000 7feb93ec3640 4 set_mon_vals no callback set 2024-03-25T18:27:53.207 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.206+0000 7feb90c5d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:53.207 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.206+0000 7feb90c5d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:53.207 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.206+0000 7feb90c5d640 10 monclient: dump: 2024-03-25T18:27:53.208 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:27:53.208 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:27:53.208 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:53.208 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:53.208 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:53.208 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:27:53.208 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:27:53.208 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:27:53.208 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:27:53.208 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:27:53.210 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.207+0000 7feb93ec3640 10 monclient: shutdown 2024-03-25T18:27:53.210 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.207+0000 7feb93ec3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:53.210 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.207+0000 7feb93ec3640 1 -- 172.21.15.155:0/2736255529 >> v1:172.21.15.155:6789/0 conn(0x7feb8c0b1730 legacy=0x7feb8c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:53.210 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.207+0000 7feb93ec3640 1 -- 172.21.15.155:0/2736255529 shutdown_connections 2024-03-25T18:27:53.210 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.207+0000 7feb93ec3640 1 -- 172.21.15.155:0/2736255529 >> 172.21.15.155:0/2736255529 conn(0x7feb8c0617a0 msgr2=0x7feb8c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:53.210 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.207+0000 7feb93ec3640 1 -- 172.21.15.155:0/2736255529 shutdown_connections 2024-03-25T18:27:53.210 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.207+0000 7feb93ec3640 1 -- 172.21.15.155:0/2736255529 wait complete. 2024-03-25T18:27:53.210 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.208+0000 7feb93ec3640 10 monclient: build_initial_monmap 2024-03-25T18:27:53.210 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.208+0000 7feb93ec3640 10 monclient: monmap: 2024-03-25T18:27:53.211 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:27:53.211 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:53.211 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:27:53.211 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:27:53.211 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:53.211 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:27:53.211 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:27:53.211 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:27:53.211 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:27:53.211 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:27:53.212 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.208+0000 7feb93ec3640 1 librados: starting msgr at 2024-03-25T18:27:53.212 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.208+0000 7feb93ec3640 1 librados: starting objecter 2024-03-25T18:27:53.212 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.208+0000 7feb93ec3640 1 Processor -- start 2024-03-25T18:27:53.212 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.209+0000 7feb93ec3640 1 -- start start 2024-03-25T18:27:53.212 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.209+0000 7feb93ec3640 1 librados: setting wanted keys 2024-03-25T18:27:53.212 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.209+0000 7feb93ec3640 1 librados: calling monclient init 2024-03-25T18:27:53.212 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.209+0000 7feb93ec3640 10 monclient: init 2024-03-25T18:27:53.213 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.209+0000 7feb93ec3640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:53.214 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.209+0000 7feb93ec3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:27:53.214 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.209+0000 7feb93ec3640 10 monclient(hunting): picked mon.noname-a con 0x7feb8c0b1730 addr v1:172.21.15.155:6789/0 2024-03-25T18:27:53.214 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.209+0000 7feb93ec3640 10 monclient(hunting): picked mon.noname-b con 0x7feb8c0b4d60 addr v1:172.21.15.167:6789/0 2024-03-25T18:27:53.214 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.210+0000 7feb93ec3640 10 monclient(hunting): picked mon.noname-c con 0x7feb8c14be90 addr v1:172.21.15.155:6790/0 2024-03-25T18:27:53.214 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.210+0000 7feb93ec3640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb8c138070 con 0x7feb8c0b1730 2024-03-25T18:27:53.214 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.210+0000 7feb93ec3640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb8c137670 con 0x7feb8c0b4d60 2024-03-25T18:27:53.214 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.210+0000 7feb93ec3640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb8c1382a0 con 0x7feb8c14be90 2024-03-25T18:27:53.215 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.210+0000 7feb93ec3640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:53.215 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.210+0000 7feb93ec3640 10 monclient(hunting): authenticate will time out at 7719.519778s 2024-03-25T18:27:53.225 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.224+0000 7feb91c5f640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7feb8c0b1730 0x7feb8c136da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:50462/0 (socket says 172.21.15.155:50462) 2024-03-25T18:27:53.225 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.224+0000 7feb91c5f640 1 -- 172.21.15.155:0/663033062 learned_addr learned my addr 172.21.15.155:0/663033062 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:27:53.225 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.224+0000 7feb827fc640 10 client.?.objecter ms_handle_connect 0x7feb8c0b1730 2024-03-25T18:27:53.225 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.224+0000 7feb827fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:53.232 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.231+0000 7feb827fc640 1 -- 172.21.15.155:0/663033062 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 14582288 0 0) 0x7feb8c138070 con 0x7feb8c0b1730 2024-03-25T18:27:53.232 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.231+0000 7feb827fc640 10 monclient(hunting): _negotiate 2024-03-25T18:27:53.232 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.231+0000 7feb827fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:53.232 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.231+0000 7feb827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:53.233 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.231+0000 7feb827fc640 10 monclient(hunting): my global_id is 4362 2024-03-25T18:27:53.233 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.231+0000 7feb827fc640 1 -- 172.21.15.155:0/663033062 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb70004eb0 con 0x7feb8c0b1730 2024-03-25T18:27:53.248 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.247+0000 7feb827fc640 10 client.?.objecter ms_handle_connect 0x7feb8c0b4d60 2024-03-25T18:27:53.248 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.247+0000 7feb827fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:53.248 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.247+0000 7feb827fc640 1 -- 172.21.15.155:0/663033062 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1028198337 0 0) 0x7feb70004eb0 con 0x7feb8c0b1730 2024-03-25T18:27:53.249 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.247+0000 7feb827fc640 1 -- 172.21.15.155:0/663033062 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb8c138070 con 0x7feb8c0b1730 2024-03-25T18:27:53.249 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.247+0000 7feb827fc640 1 -- 172.21.15.155:0/663033062 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7feb880019b0 con 0x7feb8c0b1730 2024-03-25T18:27:53.249 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.247+0000 7feb827fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:53.249 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.247+0000 7feb827fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:27:53.249 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.247+0000 7feb827fc640 10 monclient(hunting): dump: 2024-03-25T18:27:53.249 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:27:53.249 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:27:53.249 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:53.249 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:53.249 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:53.249 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:27:53.249 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:27:53.249 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:27:53.249 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:27:53.250 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:27:53.250 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.247+0000 7feb827fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:53.256 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.256+0000 7feb827fc640 1 -- 172.21.15.155:0/663033062 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 883099420 0 0) 0x7feb8c137670 con 0x7feb8c0b4d60 2024-03-25T18:27:53.257 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.256+0000 7feb827fc640 10 monclient(hunting): _negotiate 2024-03-25T18:27:53.257 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.256+0000 7feb827fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:53.257 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.256+0000 7feb827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:53.257 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.256+0000 7feb827fc640 10 monclient(hunting): my global_id is 4357 2024-03-25T18:27:53.257 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.256+0000 7feb827fc640 1 -- 172.21.15.155:0/663033062 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb70004eb0 con 0x7feb8c0b4d60 2024-03-25T18:27:53.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.260+0000 7feb827fc640 1 -- 172.21.15.155:0/663033062 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 899167145 0 0) 0x7feb8c138070 con 0x7feb8c0b1730 2024-03-25T18:27:53.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.260+0000 7feb827fc640 1 -- 172.21.15.155:0/663033062 >> v1:172.21.15.155:6790/0 conn(0x7feb8c14be90 legacy=0x7feb8c12fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:53.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.260+0000 7feb827fc640 1 -- 172.21.15.155:0/663033062 >> v1:172.21.15.167:6789/0 conn(0x7feb8c0b4d60 legacy=0x7feb8c1392f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:53.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.260+0000 7feb827fc640 10 monclient: _finish_hunting 0 2024-03-25T18:27:53.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.260+0000 7feb827fc640 1 monclient: found mon.a 2024-03-25T18:27:53.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.260+0000 7feb827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:53.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.260+0000 7feb827fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:27:53.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.260+0000 7feb827fc640 1 -- 172.21.15.155:0/663033062 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb8c138580 con 0x7feb8c0b1730 2024-03-25T18:27:53.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.260+0000 7feb827fc640 10 monclient: _finish_auth 0 2024-03-25T18:27:53.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.260+0000 7feb827fc640 10 monclient: _check_auth_tickets 2024-03-25T18:27:53.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.260+0000 7feb827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:53.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.260+0000 7feb93ec3640 5 monclient: authenticate success, global_id 4362 2024-03-25T18:27:53.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.260+0000 7feb93ec3640 10 monclient: _renew_subs 2024-03-25T18:27:53.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.260+0000 7feb93ec3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:27:53.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.260+0000 7feb93ec3640 1 -- 172.21.15.155:0/663033062 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb8c139a00 con 0x7feb8c0b1730 2024-03-25T18:27:53.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.260+0000 7feb93ec3640 10 client.4362.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:53.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.260+0000 7feb93ec3640 10 monclient: _renew_subs 2024-03-25T18:27:53.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.260+0000 7feb93ec3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:27:53.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.260+0000 7feb93ec3640 1 -- 172.21.15.155:0/663033062 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feb8c130490 con 0x7feb8c0b1730 2024-03-25T18:27:53.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.260+0000 7feb93ec3640 1 librados: init done 2024-03-25T18:27:53.264 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.261+0000 7feb93ec3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:53.264 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.261+0000 7feb93ec3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:53.264 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.261+0000 7feb93ec3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:27:53.264 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.261+0000 7feb93ec3640 1 -- 172.21.15.155:0/663033062 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb8c0627b0 con 0x7feb8c0b1730 2024-03-25T18:27:53.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.577+0000 7feb827fc640 1 -- 172.21.15.155:0/663033062 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb88001d10 con 0x7feb8c0b1730 2024-03-25T18:27:53.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.577+0000 7feb827fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:53.579 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.577+0000 7feb827fc640 1 -- 172.21.15.155:0/663033062 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7feb88003a60 con 0x7feb8c0b1730 2024-03-25T18:27:53.579 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.577+0000 7feb827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:53.579 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.577+0000 7feb827fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:53.579 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.577+0000 7feb827fc640 10 monclient: dump: 2024-03-25T18:27:53.579 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:27:53.579 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:27:53.579 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:53.579 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:27:53.579 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:53.579 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:27:53.579 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:27:53.579 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:27:53.579 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:27:53.580 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:27:53.580 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.577+0000 7feb83fff640 4 set_mon_vals no callback set 2024-03-25T18:27:53.580 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.578+0000 7feb827fc640 1 -- 172.21.15.155:0/663033062 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7feb88003c80 con 0x7feb8c0b1730 2024-03-25T18:27:53.580 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.578+0000 7feb9145e640 10 monclient: get_auth_request con 0x7feb7006dd10 auth_method 0 2024-03-25T18:27:53.580 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.578+0000 7feb827fc640 10 client.4362.objecter ms_handle_connect 0x7feb7006dd10 2024-03-25T18:27:53.585 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.583+0000 7feb827fc640 1 -- 172.21.15.155:0/663033062 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7feb88054900 con 0x7feb8c0b1730 2024-03-25T18:27:53.585 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.583+0000 7feb827fc640 10 client.4362.objecter ms_dispatch 0x7feb8c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:53.585 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.584+0000 7feb827fc640 3 client.4362.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:53.585 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.584+0000 7feb827fc640 3 client.4362.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:53.585 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.584+0000 7feb827fc640 20 client.4362.objecter dump_active .. 0 homeless 2024-03-25T18:27:53.585 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.584+0000 7feb827fc640 1 -- 172.21.15.155:0/663033062 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7feb88050410 con 0x7feb8c0b1730 2024-03-25T18:27:53.586 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.584+0000 7feb827fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:53.586 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.584+0000 7feb827fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:53.833 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.832+0000 7feb93ec3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:53.833 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.832+0000 7feb93ec3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:53.833 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.832+0000 7feb93ec3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:27:53.833 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.832+0000 7feb93ec3640 1 -- 172.21.15.155:0/663033062 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feb8c14bc70 con 0x7feb8c0b1730 2024-03-25T18:27:53.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.862+0000 7feb827fc640 1 -- 172.21.15.155:0/663033062 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7feb88052340 con 0x7feb8c0b1730 2024-03-25T18:27:53.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.862+0000 7feb827fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:53.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.862+0000 7feb827fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:53.864 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:27:53.864 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:53.866 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.865+0000 7feb93ec3640 10 librados: watch_flush enter 2024-03-25T18:27:53.866 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.865+0000 7feb93ec3640 10 librados: watch_flush exit 2024-03-25T18:27:53.866 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.865+0000 7feb93ec3640 20 client.4362.objecter shutdown clearing up homeless session... 2024-03-25T18:27:53.866 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.865+0000 7feb93ec3640 10 client.4362.objecter successfully canceled tick 2024-03-25T18:27:53.867 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.865+0000 7feb93ec3640 1 -- 172.21.15.155:0/663033062 >> v1:172.21.15.167:6800/2919070503 conn(0x7feb7006dd10 legacy=0x7feb700701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:53.867 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.865+0000 7feb93ec3640 10 monclient: shutdown 2024-03-25T18:27:53.867 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.865+0000 7feb93ec3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:53.867 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.865+0000 7feb93ec3640 1 -- 172.21.15.155:0/663033062 >> v1:172.21.15.155:6789/0 conn(0x7feb8c0b1730 legacy=0x7feb8c136da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:53.867 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.865+0000 7feb93ec3640 1 -- 172.21.15.155:0/663033062 shutdown_connections 2024-03-25T18:27:53.867 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.865+0000 7feb93ec3640 1 -- 172.21.15.155:0/663033062 >> 172.21.15.155:0/663033062 conn(0x7feb8c0617a0 msgr2=0x7feb8c13f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:53.867 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.865+0000 7feb93ec3640 1 -- 172.21.15.155:0/663033062 shutdown_connections 2024-03-25T18:27:53.867 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.865+0000 7feb93ec3640 1 -- 172.21.15.155:0/663033062 wait complete. 2024-03-25T18:27:53.867 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:27:53.865+0000 7feb93ec3640 1 librados: shutdown 2024-03-25T18:27:53.877 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:00.685 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:00.804 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.803+0000 7f0fad033640 10 monclient: get_monmap_and_config 2024-03-25T18:28:00.804 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.803+0000 7f0fad033640 10 monclient: build_initial_monmap 2024-03-25T18:28:00.804 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.803+0000 7f0fad033640 10 monclient: monmap: 2024-03-25T18:28:00.805 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:28:00.805 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:28:00.805 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:28:00.804617+0000 2024-03-25T18:28:00.805 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:28:00.804617+0000 2024-03-25T18:28:00.805 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:00.805 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:28:00.805 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:28:00.805 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:28:00.805 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:28:00.805 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:28:00.806 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.804+0000 7f0fad033640 1 Processor -- start 2024-03-25T18:28:00.806 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.804+0000 7f0fad033640 1 -- start start 2024-03-25T18:28:00.806 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.804+0000 7f0fad033640 10 monclient: init 2024-03-25T18:28:00.806 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.805+0000 7f0fad033640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:00.807 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.805+0000 7f0fad033640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:28:00.807 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.805+0000 7f0fad033640 10 monclient(hunting): picked mon.noname-a con 0x7f0fa80a98f0 addr v1:172.21.15.155:6789/0 2024-03-25T18:28:00.807 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.805+0000 7f0fad033640 10 monclient(hunting): picked mon.noname-c con 0x7f0fa813f710 addr v1:172.21.15.167:6789/0 2024-03-25T18:28:00.807 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.805+0000 7f0fad033640 10 monclient(hunting): picked mon.noname-b con 0x7f0fa81430c0 addr v1:172.21.15.155:6790/0 2024-03-25T18:28:00.807 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.805+0000 7f0fad033640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fa8142520 con 0x7f0fa80a98f0 2024-03-25T18:28:00.807 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.805+0000 7f0fad033640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fa80b28a0 con 0x7f0fa813f710 2024-03-25T18:28:00.807 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.805+0000 7f0fad033640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fa80b2a30 con 0x7f0fa81430c0 2024-03-25T18:28:00.807 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.805+0000 7f0fad033640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:00.807 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.805+0000 7f0fad033640 10 monclient(hunting): authenticate will time out at 7727.115369s 2024-03-25T18:28:00.827 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.826+0000 7f0f9ffff640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f0fa813f710 0x7f0fa81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:36304/0 (socket says 172.21.15.155:36304) 2024-03-25T18:28:00.827 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.826+0000 7f0f9ffff640 1 -- 172.21.15.155:0/330478603 learned_addr learned my addr 172.21.15.155:0/330478603 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:28:00.845 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.843+0000 7f0fa659c640 1 -- 172.21.15.155:0/330478603 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3223631241 0 0) 0x7f0fa80b28a0 con 0x7f0fa813f710 2024-03-25T18:28:00.845 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.843+0000 7f0fa659c640 10 monclient(hunting): _negotiate 2024-03-25T18:28:00.845 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.843+0000 7f0fa659c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:00.845 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.843+0000 7f0fa659c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:00.845 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.843+0000 7f0fa659c640 10 monclient(hunting): my global_id is 4360 2024-03-25T18:28:00.845 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.843+0000 7f0fa659c640 1 -- 172.21.15.155:0/330478603 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f94003520 con 0x7f0fa813f710 2024-03-25T18:28:00.848 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.847+0000 7f0fa659c640 1 -- 172.21.15.155:0/330478603 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1502193494 0 0) 0x7f0fa80b2a30 con 0x7f0fa81430c0 2024-03-25T18:28:00.848 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.847+0000 7f0fa659c640 10 monclient(hunting): _negotiate 2024-03-25T18:28:00.848 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.847+0000 7f0fa659c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:00.848 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.847+0000 7f0fa659c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:00.848 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.847+0000 7f0fa659c640 10 monclient(hunting): my global_id is 4370 2024-03-25T18:28:00.849 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.847+0000 7f0fa659c640 1 -- 172.21.15.155:0/330478603 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fa80b28a0 con 0x7f0fa81430c0 2024-03-25T18:28:00.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.852+0000 7f0fa659c640 1 -- 172.21.15.155:0/330478603 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1339519402 0 0) 0x7f0f94003520 con 0x7f0fa813f710 2024-03-25T18:28:00.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.852+0000 7f0fa659c640 1 -- 172.21.15.155:0/330478603 >> v1:172.21.15.155:6790/0 conn(0x7f0fa81430c0 legacy=0x7f0fa814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:00.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.852+0000 7f0fa659c640 1 -- 172.21.15.155:0/330478603 >> v1:172.21.15.155:6789/0 conn(0x7f0fa80a98f0 legacy=0x7f0fa80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:00.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.852+0000 7f0fa659c640 10 monclient: _finish_hunting 0 2024-03-25T18:28:00.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.852+0000 7f0fa659c640 1 monclient: found mon.noname-c 2024-03-25T18:28:00.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.852+0000 7f0fa659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:00.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.852+0000 7f0fa659c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.167:6789/0 2024-03-25T18:28:00.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.852+0000 7f0fa659c640 1 -- 172.21.15.155:0/330478603 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fa80b44c0 con 0x7f0fa813f710 2024-03-25T18:28:00.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.852+0000 7f0fa659c640 10 monclient: _finish_auth 0 2024-03-25T18:28:00.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.852+0000 7f0fa659c640 10 monclient: _check_auth_tickets 2024-03-25T18:28:00.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.852+0000 7f0fa659c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:00.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.852+0000 7f0fa659c640 1 -- 172.21.15.155:0/330478603 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f0f90001690 con 0x7f0fa813f710 2024-03-25T18:28:00.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.852+0000 7f0fad033640 5 monclient: authenticate success, global_id 4360 2024-03-25T18:28:00.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.852+0000 7f0fad033640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:00.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.852+0000 7f0fa659c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:00.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.852+0000 7f0fa659c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:28:00.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.852+0000 7f0fa659c640 10 monclient: dump: 2024-03-25T18:28:00.855 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:28:00.855 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:28:00.855 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:00.855 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:00.855 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:00.855 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:28:00.855 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:28:00.855 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:28:00.855 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:28:00.856 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:28:00.856 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.852+0000 7f0fad033640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:00.856 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.855+0000 7f0fa659c640 1 -- 172.21.15.155:0/330478603 <== mon.1 v1:172.21.15.167:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f900017f0 con 0x7f0fa813f710 2024-03-25T18:28:00.856 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.855+0000 7f0fa659c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:00.856 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.855+0000 7f0fa659c640 1 -- 172.21.15.155:0/330478603 <== mon.1 v1:172.21.15.167:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f0f90001c00 con 0x7f0fa813f710 2024-03-25T18:28:00.857 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.855+0000 7f0fa659c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:00.857 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.855+0000 7f0fa659c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:00.857 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.855+0000 7f0fa659c640 10 monclient: dump: 2024-03-25T18:28:00.857 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:28:00.857 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:28:00.857 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:00.857 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:00.857 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:00.857 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:28:00.857 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:28:00.857 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:28:00.857 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:28:00.857 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:28:00.857 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.855+0000 7f0fad033640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:00.857 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.855+0000 7f0fad033640 4 set_mon_vals no callback set 2024-03-25T18:28:00.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.856+0000 7f0fad033640 10 monclient: shutdown 2024-03-25T18:28:00.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.856+0000 7f0fad033640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:00.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.856+0000 7f0fad033640 1 -- 172.21.15.155:0/330478603 >> v1:172.21.15.167:6789/0 conn(0x7f0fa813f710 legacy=0x7f0fa81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:00.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.856+0000 7f0fad033640 1 -- 172.21.15.155:0/330478603 shutdown_connections 2024-03-25T18:28:00.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.856+0000 7f0fad033640 1 -- 172.21.15.155:0/330478603 >> 172.21.15.155:0/330478603 conn(0x7f0fa80b7ea0 msgr2=0x7f0fa80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:00.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.856+0000 7f0fad033640 1 -- 172.21.15.155:0/330478603 shutdown_connections 2024-03-25T18:28:00.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.857+0000 7f0fad033640 1 -- 172.21.15.155:0/330478603 wait complete. 2024-03-25T18:28:00.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.857+0000 7f0fad033640 10 monclient: build_initial_monmap 2024-03-25T18:28:00.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.857+0000 7f0fad033640 10 monclient: monmap: 2024-03-25T18:28:00.860 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:28:00.860 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:00.860 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:28:00.860 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:28:00.860 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:00.860 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:28:00.860 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:28:00.860 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:28:00.860 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:28:00.860 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:28:00.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.858+0000 7f0fad033640 1 librados: starting msgr at 2024-03-25T18:28:00.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.858+0000 7f0fad033640 1 librados: starting objecter 2024-03-25T18:28:00.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.858+0000 7f0fad033640 1 Processor -- start 2024-03-25T18:28:00.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.858+0000 7f0fad033640 1 -- start start 2024-03-25T18:28:00.861 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.858+0000 7f0fad033640 1 librados: setting wanted keys 2024-03-25T18:28:00.861 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.858+0000 7f0fad033640 1 librados: calling monclient init 2024-03-25T18:28:00.861 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.858+0000 7f0fad033640 10 monclient: init 2024-03-25T18:28:00.862 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.859+0000 7f0fad033640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:00.862 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.859+0000 7f0fad033640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:28:00.862 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.859+0000 7f0fad033640 10 monclient(hunting): picked mon.noname-a con 0x7f0fa80a98f0 addr v1:172.21.15.155:6789/0 2024-03-25T18:28:00.862 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.859+0000 7f0fad033640 10 monclient(hunting): picked mon.noname-c con 0x7f0fa813f710 addr v1:172.21.15.155:6790/0 2024-03-25T18:28:00.862 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.859+0000 7f0fad033640 10 monclient(hunting): picked mon.noname-b con 0x7f0fa81430c0 addr v1:172.21.15.167:6789/0 2024-03-25T18:28:00.862 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.859+0000 7f0fad033640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fa8138c10 con 0x7f0fa80a98f0 2024-03-25T18:28:00.862 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.859+0000 7f0fad033640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fa8138e60 con 0x7f0fa81430c0 2024-03-25T18:28:00.862 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.859+0000 7f0fad033640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fa8139140 con 0x7f0fa813f710 2024-03-25T18:28:00.862 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.859+0000 7f0fad033640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:00.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.859+0000 7f0fad033640 10 monclient(hunting): authenticate will time out at 7727.169039s 2024-03-25T18:28:00.868 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.867+0000 7f0f9ffff640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f0fa813f710 0x7f0fa8132440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:41682/0 (socket says 172.21.15.155:41682) 2024-03-25T18:28:00.868 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.867+0000 7f0f9ffff640 1 -- 172.21.15.155:0/1021150568 learned_addr learned my addr 172.21.15.155:0/1021150568 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:28:00.869 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.868+0000 7f0f9f7fe640 10 client.?.objecter ms_handle_connect 0x7f0fa813f710 2024-03-25T18:28:00.869 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.868+0000 7f0f9f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:00.881 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.880+0000 7f0f9f7fe640 10 client.?.objecter ms_handle_connect 0x7f0fa80a98f0 2024-03-25T18:28:00.881 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.880+0000 7f0f9f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:00.885 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.884+0000 7f0f9f7fe640 10 client.?.objecter ms_handle_connect 0x7f0fa81430c0 2024-03-25T18:28:00.885 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.884+0000 7f0f9f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:00.894 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:03:34.497 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:00.939 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.938+0000 7f0f9f7fe640 1 -- 172.21.15.155:0/1021150568 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3956731407 0 0) 0x7f0fa8138c10 con 0x7f0fa80a98f0 2024-03-25T18:28:00.939 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.938+0000 7f0f9f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:28:00.939 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.938+0000 7f0f9f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:00.939 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.938+0000 7f0f9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:00.940 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.938+0000 7f0f9f7fe640 10 monclient(hunting): my global_id is 4368 2024-03-25T18:28:00.940 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.938+0000 7f0f9f7fe640 1 -- 172.21.15.155:0/1021150568 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f88004eb0 con 0x7f0fa80a98f0 2024-03-25T18:28:00.948 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:03:34.553 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:00.975 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:03:34.579 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:00.981 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.979+0000 7f0f9f7fe640 1 -- 172.21.15.155:0/1021150568 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2265884842 0 0) 0x7f0f88004eb0 con 0x7f0fa80a98f0 2024-03-25T18:28:00.981 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.979+0000 7f0f9f7fe640 1 -- 172.21.15.155:0/1021150568 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0fa8138c10 con 0x7f0fa80a98f0 2024-03-25T18:28:00.981 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.979+0000 7f0f9f7fe640 1 -- 172.21.15.155:0/1021150568 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f0f980018c0 con 0x7f0fa80a98f0 2024-03-25T18:28:00.981 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.979+0000 7f0f9f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:00.981 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.980+0000 7f0f9f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:28:00.981 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.980+0000 7f0f9f7fe640 10 monclient(hunting): dump: 2024-03-25T18:28:00.981 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:28:00.981 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:28:00.981 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:00.981 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:00.982 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:00.982 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:28:00.982 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:28:00.982 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:28:00.982 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:28:00.982 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:28:00.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:00.980+0000 7f0f9f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:01.010 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:03:34.615 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:01.014 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.013+0000 7f0f9f7fe640 1 -- 172.21.15.155:0/1021150568 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3261049579 0 0) 0x7f0fa8139140 con 0x7f0fa813f710 2024-03-25T18:28:01.014 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.013+0000 7f0f9f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:28:01.014 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.013+0000 7f0f9f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:01.014 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.013+0000 7f0f9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:01.014 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.013+0000 7f0f9f7fe640 10 monclient(hunting): my global_id is 4373 2024-03-25T18:28:01.014 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.013+0000 7f0f9f7fe640 1 -- 172.21.15.155:0/1021150568 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f88004eb0 con 0x7f0fa813f710 2024-03-25T18:28:01.015 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:03:34.594 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:01.017 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.016+0000 7f0f9f7fe640 1 -- 172.21.15.155:0/1021150568 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1296589246 0 0) 0x7f0fa8138e60 con 0x7f0fa81430c0 2024-03-25T18:28:01.017 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.016+0000 7f0f9f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:28:01.017 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.016+0000 7f0f9f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:01.017 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.016+0000 7f0f9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:01.017 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.016+0000 7f0f9f7fe640 10 monclient(hunting): my global_id is 4363 2024-03-25T18:28:01.018 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.016+0000 7f0f9f7fe640 1 -- 172.21.15.155:0/1021150568 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fa8139140 con 0x7f0fa81430c0 2024-03-25T18:28:01.027 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:03:34.606 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:01.033 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:03:34.638 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:01.046 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.044+0000 7f0f9f7fe640 1 -- 172.21.15.155:0/1021150568 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3227385166 0 0) 0x7f0fa8138c10 con 0x7f0fa80a98f0 2024-03-25T18:28:01.046 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.044+0000 7f0f9f7fe640 1 -- 172.21.15.155:0/1021150568 >> v1:172.21.15.155:6790/0 conn(0x7f0fa813f710 legacy=0x7f0fa8132440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:01.046 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.044+0000 7f0f9f7fe640 1 -- 172.21.15.155:0/1021150568 >> v1:172.21.15.167:6789/0 conn(0x7f0fa81430c0 legacy=0x7f0fa81e34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:01.046 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.044+0000 7f0f9f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:28:01.046 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.044+0000 7f0f9f7fe640 1 monclient: found mon.a 2024-03-25T18:28:01.046 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.044+0000 7f0f9f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:01.046 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.044+0000 7f0f9f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:28:01.046 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.044+0000 7f0f9f7fe640 1 -- 172.21.15.155:0/1021150568 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fa8139920 con 0x7f0fa80a98f0 2024-03-25T18:28:01.047 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.044+0000 7f0f9f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:28:01.047 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.044+0000 7f0f9f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:28:01.047 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.044+0000 7f0f9f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:01.047 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.044+0000 7f0fad033640 5 monclient: authenticate success, global_id 4368 2024-03-25T18:28:01.047 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.044+0000 7f0fad033640 10 monclient: _renew_subs 2024-03-25T18:28:01.047 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.044+0000 7f0fad033640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:28:01.047 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.044+0000 7f0fad033640 1 -- 172.21.15.155:0/1021150568 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fa81382e0 con 0x7f0fa80a98f0 2024-03-25T18:28:01.047 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.045+0000 7f0fad033640 10 client.4368.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:01.047 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.045+0000 7f0fad033640 10 monclient: _renew_subs 2024-03-25T18:28:01.047 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.045+0000 7f0fad033640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:28:01.047 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.045+0000 7f0fad033640 1 -- 172.21.15.155:0/1021150568 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fa80ab8b0 con 0x7f0fa80a98f0 2024-03-25T18:28:01.047 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.045+0000 7f0fad033640 1 librados: init done 2024-03-25T18:28:01.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.045+0000 7f0fad033640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:01.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.045+0000 7f0fad033640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:01.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.045+0000 7f0fad033640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:28:01.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.045+0000 7f0fad033640 1 -- 172.21.15.155:0/1021150568 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fa8137a20 con 0x7f0fa80a98f0 2024-03-25T18:28:01.063 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:03:34.642 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:01.066 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:03:34.670 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:01.425 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.424+0000 7f0f9f7fe640 1 -- 172.21.15.155:0/1021150568 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f980035f0 con 0x7f0fa80a98f0 2024-03-25T18:28:01.425 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.424+0000 7f0f9f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:01.425 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.424+0000 7f0f9f7fe640 1 -- 172.21.15.155:0/1021150568 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f0f980039c0 con 0x7f0fa80a98f0 2024-03-25T18:28:01.425 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.424+0000 7f0f9f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:01.425 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.424+0000 7f0f9f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:01.426 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.424+0000 7f0f9f7fe640 10 monclient: dump: 2024-03-25T18:28:01.426 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:28:01.426 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:28:01.426 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:01.426 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:01.426 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:01.426 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:28:01.426 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:28:01.426 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:28:01.426 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:28:01.426 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:28:01.426 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.424+0000 7f0fa5d9b640 4 set_mon_vals no callback set 2024-03-25T18:28:01.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.425+0000 7f0f9f7fe640 1 -- 172.21.15.155:0/1021150568 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7f0f98003be0 con 0x7f0fa80a98f0 2024-03-25T18:28:01.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.426+0000 7f0f9ffff640 10 monclient: get_auth_request con 0x7f0f8806de80 auth_method 0 2024-03-25T18:28:01.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.426+0000 7f0f9f7fe640 10 client.4368.objecter ms_handle_connect 0x7f0f8806de80 2024-03-25T18:28:01.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.430+0000 7f0f9f7fe640 1 -- 172.21.15.155:0/1021150568 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7f0f980535f0 con 0x7f0fa80a98f0 2024-03-25T18:28:01.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.430+0000 7f0f9f7fe640 10 client.4368.objecter ms_dispatch 0x7f0fa8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:01.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.431+0000 7f0f9f7fe640 3 client.4368.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:01.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.431+0000 7f0f9f7fe640 3 client.4368.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:01.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.431+0000 7f0f9f7fe640 20 client.4368.objecter dump_active .. 0 homeless 2024-03-25T18:28:01.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.431+0000 7f0f9f7fe640 1 -- 172.21.15.155:0/1021150568 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0f9804f100 con 0x7f0fa80a98f0 2024-03-25T18:28:01.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.431+0000 7f0f9f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:01.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.431+0000 7f0f9f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:01.678 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.677+0000 7f0fad033640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:01.679 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.677+0000 7f0fad033640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:01.679 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.677+0000 7f0fad033640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:28:01.679 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.677+0000 7f0fad033640 1 -- 172.21.15.155:0/1021150568 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0fa813d010 con 0x7f0fa80a98f0 2024-03-25T18:28:01.718 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.717+0000 7f0f9f7fe640 1 -- 172.21.15.155:0/1021150568 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7f0f98051030 con 0x7f0fa80a98f0 2024-03-25T18:28:01.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.717+0000 7f0f9f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:01.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.717+0000 7f0f9f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:01.719 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:28:01.719 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:01.721 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.720+0000 7f0fad033640 10 librados: watch_flush enter 2024-03-25T18:28:01.722 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.720+0000 7f0fad033640 10 librados: watch_flush exit 2024-03-25T18:28:01.722 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.720+0000 7f0fad033640 20 client.4368.objecter shutdown clearing up homeless session... 2024-03-25T18:28:01.722 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.720+0000 7f0fad033640 10 client.4368.objecter successfully canceled tick 2024-03-25T18:28:01.722 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.720+0000 7f0fad033640 1 -- 172.21.15.155:0/1021150568 >> v1:172.21.15.167:6800/2919070503 conn(0x7f0f8806de80 legacy=0x7f0f88070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:01.722 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.720+0000 7f0fad033640 10 monclient: shutdown 2024-03-25T18:28:01.722 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.720+0000 7f0fad033640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:01.722 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.720+0000 7f0fad033640 1 -- 172.21.15.155:0/1021150568 >> v1:172.21.15.155:6789/0 conn(0x7f0fa80a98f0 legacy=0x7f0fa8137bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:01.722 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.720+0000 7f0fad033640 1 -- 172.21.15.155:0/1021150568 shutdown_connections 2024-03-25T18:28:01.722 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.720+0000 7f0fad033640 1 -- 172.21.15.155:0/1021150568 >> 172.21.15.155:0/1021150568 conn(0x7f0fa80b7ea0 msgr2=0x7f0fa8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:01.722 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.720+0000 7f0fad033640 1 -- 172.21.15.155:0/1021150568 shutdown_connections 2024-03-25T18:28:01.722 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.720+0000 7f0fad033640 1 -- 172.21.15.155:0/1021150568 wait complete. 2024-03-25T18:28:01.723 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:01.720+0000 7f0fad033640 1 librados: shutdown 2024-03-25T18:28:01.735 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:08.542 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:08.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.658+0000 7f8bd4ed0640 10 monclient: get_monmap_and_config 2024-03-25T18:28:08.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.658+0000 7f8bd4ed0640 10 monclient: build_initial_monmap 2024-03-25T18:28:08.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.658+0000 7f8bd4ed0640 10 monclient: monmap: 2024-03-25T18:28:08.660 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:28:08.660 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:28:08.660 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:28:08.659249+0000 2024-03-25T18:28:08.660 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:28:08.659249+0000 2024-03-25T18:28:08.660 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:08.660 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:28:08.660 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:28:08.660 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:28:08.660 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:28:08.660 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:28:08.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.659+0000 7f8bd4ed0640 1 Processor -- start 2024-03-25T18:28:08.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.659+0000 7f8bd4ed0640 1 -- start start 2024-03-25T18:28:08.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.659+0000 7f8bd4ed0640 10 monclient: init 2024-03-25T18:28:08.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.660+0000 7f8bd4ed0640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:08.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.660+0000 7f8bd4ed0640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:28:08.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.660+0000 7f8bd4ed0640 10 monclient(hunting): picked mon.noname-c con 0x7f8bd00b4d60 addr v1:172.21.15.167:6789/0 2024-03-25T18:28:08.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.660+0000 7f8bd4ed0640 10 monclient(hunting): picked mon.noname-b con 0x7f8bd00b1730 addr v1:172.21.15.155:6790/0 2024-03-25T18:28:08.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.660+0000 7f8bd4ed0640 10 monclient(hunting): picked mon.noname-a con 0x7f8bd014be90 addr v1:172.21.15.155:6789/0 2024-03-25T18:28:08.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.660+0000 7f8bd4ed0640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bd00af940 con 0x7f8bd014be90 2024-03-25T18:28:08.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.660+0000 7f8bd4ed0640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bd00b0a70 con 0x7f8bd00b4d60 2024-03-25T18:28:08.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.660+0000 7f8bd4ed0640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bd00b0d50 con 0x7f8bd00b1730 2024-03-25T18:28:08.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.660+0000 7f8bd4ed0640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:08.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.660+0000 7f8bd4ed0640 10 monclient(hunting): authenticate will time out at 7734.970392s 2024-03-25T18:28:08.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.665+0000 7f8bce59c640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f8bd00b4d60 0x7f8bd013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:60460/0 (socket says 172.21.15.155:60460) 2024-03-25T18:28:08.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.665+0000 7f8bce59c640 1 -- 172.21.15.155:0/3350861170 learned_addr learned my addr 172.21.15.155:0/3350861170 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:28:08.682 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.680+0000 7f8bcd59a640 1 -- 172.21.15.155:0/3350861170 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2755239337 0 0) 0x7f8bd00b0a70 con 0x7f8bd00b4d60 2024-03-25T18:28:08.682 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.680+0000 7f8bcd59a640 10 monclient(hunting): _negotiate 2024-03-25T18:28:08.682 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.680+0000 7f8bcd59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:08.683 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.680+0000 7f8bcd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:08.683 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.680+0000 7f8bcd59a640 10 monclient(hunting): my global_id is 4366 2024-03-25T18:28:08.683 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.680+0000 7f8bcd59a640 1 -- 172.21.15.155:0/3350861170 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bb8003520 con 0x7f8bd00b4d60 2024-03-25T18:28:08.690 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.689+0000 7f8bcd59a640 1 -- 172.21.15.155:0/3350861170 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1590798805 0 0) 0x7f8bd00af940 con 0x7f8bd014be90 2024-03-25T18:28:08.690 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.689+0000 7f8bcd59a640 10 monclient(hunting): _negotiate 2024-03-25T18:28:08.690 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.689+0000 7f8bcd59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:08.690 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.689+0000 7f8bcd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:08.690 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.689+0000 7f8bcd59a640 10 monclient(hunting): my global_id is 4371 2024-03-25T18:28:08.690 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.689+0000 7f8bcd59a640 1 -- 172.21.15.155:0/3350861170 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bd00b0a70 con 0x7f8bd014be90 2024-03-25T18:28:08.700 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.699+0000 7f8bcd59a640 1 -- 172.21.15.155:0/3350861170 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2902671634 0 0) 0x7f8bd00b0a70 con 0x7f8bd014be90 2024-03-25T18:28:08.700 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.699+0000 7f8bcd59a640 1 -- 172.21.15.155:0/3350861170 >> v1:172.21.15.155:6790/0 conn(0x7f8bd00b1730 legacy=0x7f8bd00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:08.700 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.699+0000 7f8bcd59a640 1 -- 172.21.15.155:0/3350861170 >> v1:172.21.15.167:6789/0 conn(0x7f8bd00b4d60 legacy=0x7f8bd013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:08.701 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.699+0000 7f8bcd59a640 10 monclient: _finish_hunting 0 2024-03-25T18:28:08.701 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.699+0000 7f8bcd59a640 1 monclient: found mon.noname-a 2024-03-25T18:28:08.701 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.699+0000 7f8bcd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:08.701 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.699+0000 7f8bcd59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-25T18:28:08.701 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.699+0000 7f8bcd59a640 1 -- 172.21.15.155:0/3350861170 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bd00b1030 con 0x7f8bd014be90 2024-03-25T18:28:08.701 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.699+0000 7f8bcd59a640 10 monclient: _finish_auth 0 2024-03-25T18:28:08.701 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.699+0000 7f8bcd59a640 10 monclient: _check_auth_tickets 2024-03-25T18:28:08.701 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.699+0000 7f8bcd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:08.701 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.699+0000 7f8bcd59a640 1 -- 172.21.15.155:0/3350861170 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f8bc80019f0 con 0x7f8bd014be90 2024-03-25T18:28:08.701 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.699+0000 7f8bcd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:08.701 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.699+0000 7f8bcd59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:28:08.701 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.699+0000 7f8bcd59a640 10 monclient: dump: 2024-03-25T18:28:08.701 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:28:08.701 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:28:08.702 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:08.702 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:08.702 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:08.702 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:28:08.702 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:28:08.702 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:28:08.702 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:28:08.702 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:28:08.702 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.699+0000 7f8bd4ed0640 5 monclient: authenticate success, global_id 4371 2024-03-25T18:28:08.702 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.699+0000 7f8bd4ed0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:08.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.723+0000 7f8bcd59a640 1 -- 172.21.15.155:0/3350861170 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8bc8001b50 con 0x7f8bd014be90 2024-03-25T18:28:08.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.723+0000 7f8bcd59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:08.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.723+0000 7f8bcd59a640 1 -- 172.21.15.155:0/3350861170 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f8bc8001f60 con 0x7f8bd014be90 2024-03-25T18:28:08.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.723+0000 7f8bd4ed0640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:08.725 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.723+0000 7f8bd4ed0640 4 set_mon_vals no callback set 2024-03-25T18:28:08.725 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.723+0000 7f8bcd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:08.725 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.723+0000 7f8bcd59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:08.725 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.723+0000 7f8bcd59a640 10 monclient: dump: 2024-03-25T18:28:08.725 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:28:08.725 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:28:08.725 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:08.725 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:08.725 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:08.725 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:28:08.725 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:28:08.725 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:28:08.725 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:28:08.726 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:28:08.726 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.724+0000 7f8bd4ed0640 10 monclient: shutdown 2024-03-25T18:28:08.727 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.724+0000 7f8bd4ed0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:08.727 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.724+0000 7f8bd4ed0640 1 -- 172.21.15.155:0/3350861170 >> v1:172.21.15.155:6789/0 conn(0x7f8bd014be90 legacy=0x7f8bd014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:08.727 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.724+0000 7f8bd4ed0640 1 -- 172.21.15.155:0/3350861170 shutdown_connections 2024-03-25T18:28:08.727 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.724+0000 7f8bd4ed0640 1 -- 172.21.15.155:0/3350861170 >> 172.21.15.155:0/3350861170 conn(0x7f8bd00617a0 msgr2=0x7f8bd0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:08.727 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.725+0000 7f8bd4ed0640 1 -- 172.21.15.155:0/3350861170 shutdown_connections 2024-03-25T18:28:08.727 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.725+0000 7f8bd4ed0640 1 -- 172.21.15.155:0/3350861170 wait complete. 2024-03-25T18:28:08.727 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.726+0000 7f8bd4ed0640 10 monclient: build_initial_monmap 2024-03-25T18:28:08.727 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.726+0000 7f8bd4ed0640 10 monclient: monmap: 2024-03-25T18:28:08.727 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:28:08.727 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:08.727 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:28:08.728 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:28:08.728 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:08.728 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:28:08.728 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:28:08.728 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:28:08.728 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:28:08.728 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:28:08.729 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.726+0000 7f8bd4ed0640 1 librados: starting msgr at 2024-03-25T18:28:08.729 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.726+0000 7f8bd4ed0640 1 librados: starting objecter 2024-03-25T18:28:08.729 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.726+0000 7f8bd4ed0640 1 Processor -- start 2024-03-25T18:28:08.729 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.726+0000 7f8bd4ed0640 1 -- start start 2024-03-25T18:28:08.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.726+0000 7f8bd4ed0640 1 librados: setting wanted keys 2024-03-25T18:28:08.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.726+0000 7f8bd4ed0640 1 librados: calling monclient init 2024-03-25T18:28:08.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.726+0000 7f8bd4ed0640 10 monclient: init 2024-03-25T18:28:08.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.727+0000 7f8bd4ed0640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:08.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.727+0000 7f8bd4ed0640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:28:08.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.727+0000 7f8bd4ed0640 10 monclient(hunting): picked mon.noname-c con 0x7f8bd00b1730 addr v1:172.21.15.155:6790/0 2024-03-25T18:28:08.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.727+0000 7f8bd4ed0640 10 monclient(hunting): picked mon.noname-b con 0x7f8bd00b4d60 addr v1:172.21.15.167:6789/0 2024-03-25T18:28:08.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.727+0000 7f8bd4ed0640 10 monclient(hunting): picked mon.noname-a con 0x7f8bd014be90 addr v1:172.21.15.155:6789/0 2024-03-25T18:28:08.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.727+0000 7f8bd4ed0640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bd01d7690 con 0x7f8bd014be90 2024-03-25T18:28:08.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.727+0000 7f8bd4ed0640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bd0142ff0 con 0x7f8bd00b4d60 2024-03-25T18:28:08.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.727+0000 7f8bd4ed0640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bd0143900 con 0x7f8bd00b1730 2024-03-25T18:28:08.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.727+0000 7f8bd4ed0640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:08.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.727+0000 7f8bd4ed0640 10 monclient(hunting): authenticate will time out at 7735.037383s 2024-03-25T18:28:08.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.730+0000 7f8bced9d640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f8bd014be90 0x7f8bd01e20d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:56918/0 (socket says 172.21.15.155:56918) 2024-03-25T18:28:08.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.730+0000 7f8bced9d640 1 -- 172.21.15.155:0/4174350519 learned_addr learned my addr 172.21.15.155:0/4174350519 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:28:08.732 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.731+0000 7f8bbeffd640 10 client.?.objecter ms_handle_connect 0x7f8bd014be90 2024-03-25T18:28:08.732 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.731+0000 7f8bbeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:08.749 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.748+0000 7f8bbeffd640 1 -- 172.21.15.155:0/4174350519 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3222307874 0 0) 0x7f8bd01d7690 con 0x7f8bd014be90 2024-03-25T18:28:08.749 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.748+0000 7f8bbeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:28:08.749 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.748+0000 7f8bbeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:08.749 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.748+0000 7f8bbeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:08.749 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.748+0000 7f8bbeffd640 10 monclient(hunting): my global_id is 4374 2024-03-25T18:28:08.749 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.748+0000 7f8bbeffd640 1 -- 172.21.15.155:0/4174350519 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bb0004eb0 con 0x7f8bd014be90 2024-03-25T18:28:08.756 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.756+0000 7f8bbeffd640 10 client.?.objecter ms_handle_connect 0x7f8bd00b4d60 2024-03-25T18:28:08.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.756+0000 7f8bbeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:08.759 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.758+0000 7f8bbeffd640 10 client.?.objecter ms_handle_connect 0x7f8bd00b1730 2024-03-25T18:28:08.759 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.758+0000 7f8bbeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:08.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.766+0000 7f8bbeffd640 1 -- 172.21.15.155:0/4174350519 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3644962393 0 0) 0x7f8bd0142ff0 con 0x7f8bd00b4d60 2024-03-25T18:28:08.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.766+0000 7f8bbeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:28:08.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.766+0000 7f8bbeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:08.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.766+0000 7f8bbeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:08.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.766+0000 7f8bbeffd640 10 monclient(hunting): my global_id is 4369 2024-03-25T18:28:08.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.766+0000 7f8bbeffd640 1 -- 172.21.15.155:0/4174350519 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bd01d7690 con 0x7f8bd00b4d60 2024-03-25T18:28:08.772 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.771+0000 7f8bbeffd640 1 -- 172.21.15.155:0/4174350519 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1929701447 0 0) 0x7f8bb0004eb0 con 0x7f8bd014be90 2024-03-25T18:28:08.772 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.771+0000 7f8bbeffd640 1 -- 172.21.15.155:0/4174350519 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8bd0142ff0 con 0x7f8bd014be90 2024-03-25T18:28:08.772 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.771+0000 7f8bbeffd640 1 -- 172.21.15.155:0/4174350519 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f8bc8001f60 con 0x7f8bd014be90 2024-03-25T18:28:08.772 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.771+0000 7f8bbeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:08.772 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.771+0000 7f8bbeffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:28:08.772 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.771+0000 7f8bbeffd640 10 monclient(hunting): dump: 2024-03-25T18:28:08.772 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:28:08.772 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:28:08.773 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:08.773 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:08.773 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:08.773 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:28:08.773 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:28:08.773 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:28:08.773 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:28:08.773 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:28:08.773 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.771+0000 7f8bbeffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:08.773 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.771+0000 7f8bbeffd640 1 -- 172.21.15.155:0/4174350519 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2874537777 0 0) 0x7f8bd0143900 con 0x7f8bd00b1730 2024-03-25T18:28:08.773 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.771+0000 7f8bbeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:28:08.773 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.771+0000 7f8bbeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:08.773 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.771+0000 7f8bbeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:08.773 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.771+0000 7f8bbeffd640 10 monclient(hunting): my global_id is 4379 2024-03-25T18:28:08.773 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.771+0000 7f8bbeffd640 1 -- 172.21.15.155:0/4174350519 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bb0004eb0 con 0x7f8bd00b1730 2024-03-25T18:28:08.779 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.778+0000 7f8bbeffd640 1 -- 172.21.15.155:0/4174350519 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2160862381 0 0) 0x7f8bd0142ff0 con 0x7f8bd014be90 2024-03-25T18:28:08.779 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.778+0000 7f8bbeffd640 1 -- 172.21.15.155:0/4174350519 >> v1:172.21.15.155:6790/0 conn(0x7f8bd00b1730 legacy=0x7f8bd0141510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:08.779 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.778+0000 7f8bbeffd640 1 -- 172.21.15.155:0/4174350519 >> v1:172.21.15.167:6789/0 conn(0x7f8bd00b4d60 legacy=0x7f8bd01d7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:08.779 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.778+0000 7f8bbeffd640 10 monclient: _finish_hunting 0 2024-03-25T18:28:08.779 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.778+0000 7f8bbeffd640 1 monclient: found mon.a 2024-03-25T18:28:08.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.778+0000 7f8bbeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:08.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.778+0000 7f8bbeffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:28:08.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.778+0000 7f8bbeffd640 1 -- 172.21.15.155:0/4174350519 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bd0140990 con 0x7f8bd014be90 2024-03-25T18:28:08.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.778+0000 7f8bbeffd640 10 monclient: _finish_auth 0 2024-03-25T18:28:08.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.778+0000 7f8bbeffd640 10 monclient: _check_auth_tickets 2024-03-25T18:28:08.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.778+0000 7f8bbeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:08.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.778+0000 7f8bd4ed0640 5 monclient: authenticate success, global_id 4374 2024-03-25T18:28:08.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.778+0000 7f8bd4ed0640 10 monclient: _renew_subs 2024-03-25T18:28:08.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.778+0000 7f8bd4ed0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:28:08.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.778+0000 7f8bd4ed0640 1 -- 172.21.15.155:0/4174350519 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bd0141c20 con 0x7f8bd014be90 2024-03-25T18:28:08.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.778+0000 7f8bd4ed0640 10 client.4374.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:08.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.778+0000 7f8bd4ed0640 10 monclient: _renew_subs 2024-03-25T18:28:08.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.778+0000 7f8bd4ed0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:28:08.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.778+0000 7f8bd4ed0640 1 -- 172.21.15.155:0/4174350519 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bd01d8480 con 0x7f8bd014be90 2024-03-25T18:28:08.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.778+0000 7f8bd4ed0640 1 librados: init done 2024-03-25T18:28:08.781 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.779+0000 7f8bd4ed0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:08.781 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.779+0000 7f8bd4ed0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:08.781 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.779+0000 7f8bd4ed0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:28:08.781 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:08.779+0000 7f8bd4ed0640 1 -- 172.21.15.155:0/4174350519 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bd00627b0 con 0x7f8bd014be90 2024-03-25T18:28:09.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:09.090+0000 7f8bbeffd640 1 -- 172.21.15.155:0/4174350519 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8bc8003c20 con 0x7f8bd014be90 2024-03-25T18:28:09.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:09.090+0000 7f8bbeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:09.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:09.090+0000 7f8bbeffd640 1 -- 172.21.15.155:0/4174350519 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f8bc8003ff0 con 0x7f8bd014be90 2024-03-25T18:28:09.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:09.090+0000 7f8bbeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:09.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:09.091+0000 7f8bbeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:09.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:09.091+0000 7f8bbeffd640 10 monclient: dump: 2024-03-25T18:28:09.092 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:28:09.092 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:28:09.092 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:09.093 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:09.093 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:09.093 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:28:09.093 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:28:09.093 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:28:09.093 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:28:09.093 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:28:09.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:09.091+0000 7f8bccd99640 4 set_mon_vals no callback set 2024-03-25T18:28:09.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:09.091+0000 7f8bbeffd640 1 -- 172.21.15.155:0/4174350519 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7f8bc8004210 con 0x7f8bd014be90 2024-03-25T18:28:09.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:09.092+0000 7f8bce59c640 10 monclient: get_auth_request con 0x7f8bb006de20 auth_method 0 2024-03-25T18:28:09.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:09.092+0000 7f8bbeffd640 10 client.4374.objecter ms_handle_connect 0x7f8bb006de20 2024-03-25T18:28:09.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:09.103+0000 7f8bbeffd640 1 -- 172.21.15.155:0/4174350519 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7f8bc8054ac0 con 0x7f8bd014be90 2024-03-25T18:28:09.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:09.103+0000 7f8bbeffd640 10 client.4374.objecter ms_dispatch 0x7f8bd0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:09.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:09.103+0000 7f8bbeffd640 3 client.4374.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:09.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:09.103+0000 7f8bbeffd640 3 client.4374.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:09.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:09.103+0000 7f8bbeffd640 20 client.4374.objecter dump_active .. 0 homeless 2024-03-25T18:28:09.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:09.103+0000 7f8bbeffd640 1 -- 172.21.15.155:0/4174350519 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8bc80505d0 con 0x7f8bd014be90 2024-03-25T18:28:09.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:09.104+0000 7f8bbeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:09.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:09.104+0000 7f8bbeffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:09.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:09.322+0000 7f8bd4ed0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:09.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:09.322+0000 7f8bd4ed0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:09.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:09.322+0000 7f8bd4ed0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:28:09.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:09.322+0000 7f8bd4ed0640 1 -- 172.21.15.155:0/4174350519 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8bd0140b20 con 0x7f8bd014be90 2024-03-25T18:28:09.366 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:09.365+0000 7f8bbeffd640 1 -- 172.21.15.155:0/4174350519 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7f8bc8052500 con 0x7f8bd014be90 2024-03-25T18:28:09.366 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:09.365+0000 7f8bbeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:09.367 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:09.365+0000 7f8bbeffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:09.367 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:28:09.367 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:09.369 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:09.367+0000 7f8bd4ed0640 10 librados: watch_flush enter 2024-03-25T18:28:09.369 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:09.368+0000 7f8bd4ed0640 10 librados: watch_flush exit 2024-03-25T18:28:09.369 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:09.368+0000 7f8bd4ed0640 20 client.4374.objecter shutdown clearing up homeless session... 2024-03-25T18:28:09.369 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:09.368+0000 7f8bd4ed0640 10 client.4374.objecter successfully canceled tick 2024-03-25T18:28:09.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:09.368+0000 7f8bd4ed0640 1 -- 172.21.15.155:0/4174350519 >> v1:172.21.15.167:6800/2919070503 conn(0x7f8bb006de20 legacy=0x7f8bb00702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:09.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:09.368+0000 7f8bd4ed0640 10 monclient: shutdown 2024-03-25T18:28:09.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:09.368+0000 7f8bd4ed0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:09.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:09.368+0000 7f8bd4ed0640 1 -- 172.21.15.155:0/4174350519 >> v1:172.21.15.155:6789/0 conn(0x7f8bd014be90 legacy=0x7f8bd01e20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:09.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:09.368+0000 7f8bd4ed0640 1 -- 172.21.15.155:0/4174350519 shutdown_connections 2024-03-25T18:28:09.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:09.368+0000 7f8bd4ed0640 1 -- 172.21.15.155:0/4174350519 >> 172.21.15.155:0/4174350519 conn(0x7f8bd00617a0 msgr2=0x7f8bd0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:09.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:09.368+0000 7f8bd4ed0640 1 -- 172.21.15.155:0/4174350519 shutdown_connections 2024-03-25T18:28:09.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:09.368+0000 7f8bd4ed0640 1 -- 172.21.15.155:0/4174350519 wait complete. 2024-03-25T18:28:09.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:09.368+0000 7f8bd4ed0640 1 librados: shutdown 2024-03-25T18:28:09.382 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:16.190 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:16.306 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.305+0000 7f0c49fc8640 10 monclient: get_monmap_and_config 2024-03-25T18:28:16.306 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.305+0000 7f0c49fc8640 10 monclient: build_initial_monmap 2024-03-25T18:28:16.306 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.305+0000 7f0c49fc8640 10 monclient: monmap: 2024-03-25T18:28:16.306 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:28:16.306 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:28:16.306 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:28:16.306050+0000 2024-03-25T18:28:16.306 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:28:16.306050+0000 2024-03-25T18:28:16.306 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:16.307 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:28:16.307 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:28:16.307 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:28:16.307 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:28:16.307 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:28:16.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.305+0000 7f0c49fc8640 1 Processor -- start 2024-03-25T18:28:16.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.306+0000 7f0c49fc8640 1 -- start start 2024-03-25T18:28:16.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.306+0000 7f0c49fc8640 10 monclient: init 2024-03-25T18:28:16.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.306+0000 7f0c49fc8640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:16.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.306+0000 7f0c49fc8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:28:16.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.306+0000 7f0c49fc8640 10 monclient(hunting): picked mon.noname-c con 0x7f0c440b4d60 addr v1:172.21.15.167:6789/0 2024-03-25T18:28:16.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.306+0000 7f0c49fc8640 10 monclient(hunting): picked mon.noname-b con 0x7f0c440b1730 addr v1:172.21.15.155:6790/0 2024-03-25T18:28:16.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.306+0000 7f0c49fc8640 10 monclient(hunting): picked mon.noname-a con 0x7f0c4414be90 addr v1:172.21.15.155:6789/0 2024-03-25T18:28:16.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.306+0000 7f0c49fc8640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c440af940 con 0x7f0c4414be90 2024-03-25T18:28:16.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.306+0000 7f0c49fc8640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c440b0a70 con 0x7f0c440b4d60 2024-03-25T18:28:16.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.307+0000 7f0c49fc8640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c440b0d50 con 0x7f0c440b1730 2024-03-25T18:28:16.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.307+0000 7f0c49fc8640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:16.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.307+0000 7f0c49fc8640 10 monclient(hunting): authenticate will time out at 7742.616726s 2024-03-25T18:28:16.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.314+0000 7f0c437fe640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f0c440b4d60 0x7f0c4413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:37116/0 (socket says 172.21.15.155:37116) 2024-03-25T18:28:16.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.314+0000 7f0c437fe640 1 -- 172.21.15.155:0/2017647382 learned_addr learned my addr 172.21.15.155:0/2017647382 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:28:16.327 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.326+0000 7f0c427fc640 1 -- 172.21.15.155:0/2017647382 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1553453831 0 0) 0x7f0c440b0a70 con 0x7f0c440b4d60 2024-03-25T18:28:16.327 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.326+0000 7f0c427fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:16.327 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.326+0000 7f0c427fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:16.327 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.326+0000 7f0c427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:16.327 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.326+0000 7f0c427fc640 10 monclient(hunting): my global_id is 4372 2024-03-25T18:28:16.327 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.326+0000 7f0c427fc640 1 -- 172.21.15.155:0/2017647382 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c2c003520 con 0x7f0c440b4d60 2024-03-25T18:28:16.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.329+0000 7f0c427fc640 1 -- 172.21.15.155:0/2017647382 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2528522074 0 0) 0x7f0c440b0d50 con 0x7f0c440b1730 2024-03-25T18:28:16.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.329+0000 7f0c427fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:16.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.329+0000 7f0c427fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:16.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.329+0000 7f0c427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:16.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.329+0000 7f0c427fc640 10 monclient(hunting): my global_id is 4382 2024-03-25T18:28:16.331 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.329+0000 7f0c427fc640 1 -- 172.21.15.155:0/2017647382 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c440b0a70 con 0x7f0c440b1730 2024-03-25T18:28:16.335 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.334+0000 7f0c427fc640 1 -- 172.21.15.155:0/2017647382 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 744150737 0 0) 0x7f0c440af940 con 0x7f0c4414be90 2024-03-25T18:28:16.335 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.334+0000 7f0c427fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:16.335 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.334+0000 7f0c427fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:16.336 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.334+0000 7f0c427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:16.336 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.334+0000 7f0c427fc640 10 monclient(hunting): my global_id is 4377 2024-03-25T18:28:16.336 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.334+0000 7f0c427fc640 1 -- 172.21.15.155:0/2017647382 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c440b0d50 con 0x7f0c4414be90 2024-03-25T18:28:16.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.340+0000 7f0c427fc640 1 -- 172.21.15.155:0/2017647382 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3588706973 0 0) 0x7f0c440b0a70 con 0x7f0c440b1730 2024-03-25T18:28:16.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.340+0000 7f0c427fc640 1 -- 172.21.15.155:0/2017647382 >> v1:172.21.15.167:6789/0 conn(0x7f0c440b4d60 legacy=0x7f0c4413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:16.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.341+0000 7f0c427fc640 1 -- 172.21.15.155:0/2017647382 >> v1:172.21.15.155:6789/0 conn(0x7f0c4414be90 legacy=0x7f0c4414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:16.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.341+0000 7f0c427fc640 10 monclient: _finish_hunting 0 2024-03-25T18:28:16.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.341+0000 7f0c427fc640 1 monclient: found mon.noname-b 2024-03-25T18:28:16.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.341+0000 7f0c427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:16.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.341+0000 7f0c427fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-25T18:28:16.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.341+0000 7f0c427fc640 1 -- 172.21.15.155:0/2017647382 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c440b1030 con 0x7f0c440b1730 2024-03-25T18:28:16.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.341+0000 7f0c427fc640 10 monclient: _finish_auth 0 2024-03-25T18:28:16.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.341+0000 7f0c427fc640 10 monclient: _check_auth_tickets 2024-03-25T18:28:16.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.341+0000 7f0c427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:16.343 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.341+0000 7f0c427fc640 1 -- 172.21.15.155:0/2017647382 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f0c38001670 con 0x7f0c440b1730 2024-03-25T18:28:16.343 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.341+0000 7f0c427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:16.343 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.341+0000 7f0c427fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:28:16.343 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.341+0000 7f0c427fc640 10 monclient: dump: 2024-03-25T18:28:16.343 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:28:16.343 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:28:16.343 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:16.343 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:16.343 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:16.343 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:28:16.343 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:28:16.343 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:28:16.343 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:28:16.343 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:28:16.343 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.341+0000 7f0c49fc8640 5 monclient: authenticate success, global_id 4382 2024-03-25T18:28:16.344 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.341+0000 7f0c49fc8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:16.368 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.367+0000 7f0c427fc640 1 -- 172.21.15.155:0/2017647382 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0c380017d0 con 0x7f0c440b1730 2024-03-25T18:28:16.368 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.367+0000 7f0c427fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:16.368 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.367+0000 7f0c427fc640 1 -- 172.21.15.155:0/2017647382 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f0c38001be0 con 0x7f0c440b1730 2024-03-25T18:28:16.368 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.367+0000 7f0c427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:16.369 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.367+0000 7f0c427fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:16.369 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.367+0000 7f0c427fc640 10 monclient: dump: 2024-03-25T18:28:16.369 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:28:16.369 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:28:16.369 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:16.369 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:16.369 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:16.369 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:28:16.369 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:28:16.369 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:28:16.369 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:28:16.369 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:28:16.369 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.367+0000 7f0c49fc8640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:16.369 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.367+0000 7f0c49fc8640 4 set_mon_vals no callback set 2024-03-25T18:28:16.371 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.368+0000 7f0c49fc8640 10 monclient: shutdown 2024-03-25T18:28:16.371 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.368+0000 7f0c49fc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:16.371 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.368+0000 7f0c49fc8640 1 -- 172.21.15.155:0/2017647382 >> v1:172.21.15.155:6790/0 conn(0x7f0c440b1730 legacy=0x7f0c440b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:16.371 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.368+0000 7f0c49fc8640 1 -- 172.21.15.155:0/2017647382 shutdown_connections 2024-03-25T18:28:16.371 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.368+0000 7f0c49fc8640 1 -- 172.21.15.155:0/2017647382 >> 172.21.15.155:0/2017647382 conn(0x7f0c440617a0 msgr2=0x7f0c44061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:16.371 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.368+0000 7f0c49fc8640 1 -- 172.21.15.155:0/2017647382 shutdown_connections 2024-03-25T18:28:16.371 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.368+0000 7f0c49fc8640 1 -- 172.21.15.155:0/2017647382 wait complete. 2024-03-25T18:28:16.371 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.369+0000 7f0c49fc8640 10 monclient: build_initial_monmap 2024-03-25T18:28:16.371 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.369+0000 7f0c49fc8640 10 monclient: monmap: 2024-03-25T18:28:16.371 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:28:16.371 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:16.371 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:28:16.372 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:28:16.372 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:16.372 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:28:16.372 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:28:16.372 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:28:16.372 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:28:16.372 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:28:16.372 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.369+0000 7f0c49fc8640 1 librados: starting msgr at 2024-03-25T18:28:16.372 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.369+0000 7f0c49fc8640 1 librados: starting objecter 2024-03-25T18:28:16.372 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.369+0000 7f0c49fc8640 1 Processor -- start 2024-03-25T18:28:16.372 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.370+0000 7f0c49fc8640 1 -- start start 2024-03-25T18:28:16.372 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.370+0000 7f0c49fc8640 1 librados: setting wanted keys 2024-03-25T18:28:16.372 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.370+0000 7f0c49fc8640 1 librados: calling monclient init 2024-03-25T18:28:16.372 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.370+0000 7f0c49fc8640 10 monclient: init 2024-03-25T18:28:16.373 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.370+0000 7f0c49fc8640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:16.373 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.370+0000 7f0c49fc8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:28:16.373 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.370+0000 7f0c49fc8640 10 monclient(hunting): picked mon.noname-b con 0x7f0c440b1730 addr v1:172.21.15.167:6789/0 2024-03-25T18:28:16.374 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.370+0000 7f0c49fc8640 10 monclient(hunting): picked mon.noname-a con 0x7f0c440b4d60 addr v1:172.21.15.155:6789/0 2024-03-25T18:28:16.374 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.370+0000 7f0c49fc8640 10 monclient(hunting): picked mon.noname-c con 0x7f0c4414be90 addr v1:172.21.15.155:6790/0 2024-03-25T18:28:16.374 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.370+0000 7f0c49fc8640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c440a9610 con 0x7f0c440b4d60 2024-03-25T18:28:16.374 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.370+0000 7f0c49fc8640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c441433d0 con 0x7f0c440b1730 2024-03-25T18:28:16.374 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.370+0000 7f0c49fc8640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c44143560 con 0x7f0c4414be90 2024-03-25T18:28:16.374 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.370+0000 7f0c49fc8640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:16.374 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.370+0000 7f0c49fc8640 10 monclient(hunting): authenticate will time out at 7742.680687s 2024-03-25T18:28:16.382 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.381+0000 7f0c437fe640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f0c440b1730 0x7f0c441d79a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:37196/0 (socket says 172.21.15.155:37196) 2024-03-25T18:28:16.382 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.381+0000 7f0c437fe640 1 -- 172.21.15.155:0/3523729997 learned_addr learned my addr 172.21.15.155:0/3523729997 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:28:16.383 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.382+0000 7f0c2bfff640 10 client.?.objecter ms_handle_connect 0x7f0c440b1730 2024-03-25T18:28:16.383 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.382+0000 7f0c2bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:16.386 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.386+0000 7f0c2bfff640 10 client.?.objecter ms_handle_connect 0x7f0c4414be90 2024-03-25T18:28:16.386 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.386+0000 7f0c2bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:16.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.397+0000 7f0c2bfff640 1 -- 172.21.15.155:0/3523729997 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1289477322 0 0) 0x7f0c44143560 con 0x7f0c4414be90 2024-03-25T18:28:16.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.398+0000 7f0c2bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:28:16.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.398+0000 7f0c2bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:16.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.398+0000 7f0c2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:16.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.398+0000 7f0c2bfff640 10 monclient(hunting): my global_id is 4385 2024-03-25T18:28:16.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.398+0000 7f0c2bfff640 1 -- 172.21.15.155:0/3523729997 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c30004eb0 con 0x7f0c4414be90 2024-03-25T18:28:16.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.398+0000 7f0c2bfff640 1 -- 172.21.15.155:0/3523729997 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3425171234 0 0) 0x7f0c441433d0 con 0x7f0c440b1730 2024-03-25T18:28:16.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.398+0000 7f0c2bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:28:16.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.398+0000 7f0c2bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:16.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.398+0000 7f0c2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:16.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.398+0000 7f0c2bfff640 10 monclient(hunting): my global_id is 4375 2024-03-25T18:28:16.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.398+0000 7f0c2bfff640 1 -- 172.21.15.155:0/3523729997 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c44143560 con 0x7f0c440b1730 2024-03-25T18:28:16.403 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.403+0000 7f0c2bfff640 10 client.?.objecter ms_handle_connect 0x7f0c440b4d60 2024-03-25T18:28:16.403 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.403+0000 7f0c2bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:16.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.409+0000 7f0c2bfff640 1 -- 172.21.15.155:0/3523729997 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3132594396 0 0) 0x7f0c440a9610 con 0x7f0c440b4d60 2024-03-25T18:28:16.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.409+0000 7f0c2bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:28:16.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.409+0000 7f0c2bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:16.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.409+0000 7f0c2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:16.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.409+0000 7f0c2bfff640 10 monclient(hunting): my global_id is 4380 2024-03-25T18:28:16.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.409+0000 7f0c2bfff640 1 -- 172.21.15.155:0/3523729997 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c441433d0 con 0x7f0c440b4d60 2024-03-25T18:28:16.422 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.420+0000 7f0c2bfff640 1 -- 172.21.15.155:0/3523729997 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1122044995 0 0) 0x7f0c44143560 con 0x7f0c440b1730 2024-03-25T18:28:16.422 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.420+0000 7f0c2bfff640 1 -- 172.21.15.155:0/3523729997 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0c440a9610 con 0x7f0c440b1730 2024-03-25T18:28:16.422 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.421+0000 7f0c2bfff640 1 -- 172.21.15.155:0/3523729997 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f0c34001950 con 0x7f0c440b1730 2024-03-25T18:28:16.422 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.421+0000 7f0c2bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:16.422 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.421+0000 7f0c2bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:28:16.422 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.421+0000 7f0c2bfff640 10 monclient(hunting): dump: 2024-03-25T18:28:16.422 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:28:16.422 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:28:16.422 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:16.422 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:16.422 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:16.422 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:28:16.422 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:28:16.422 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:28:16.423 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:28:16.423 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:28:16.423 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.421+0000 7f0c2bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:16.423 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.421+0000 7f0c2bfff640 1 -- 172.21.15.155:0/3523729997 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2493693344 0 0) 0x7f0c30004eb0 con 0x7f0c4414be90 2024-03-25T18:28:16.423 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.421+0000 7f0c2bfff640 1 -- 172.21.15.155:0/3523729997 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0c44143560 con 0x7f0c4414be90 2024-03-25T18:28:16.423 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.421+0000 7f0c2bfff640 1 -- 172.21.15.155:0/3523729997 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f0c3c001bd0 con 0x7f0c4414be90 2024-03-25T18:28:16.423 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.421+0000 7f0c2bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:16.424 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.421+0000 7f0c2bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:16.424 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.421+0000 7f0c2bfff640 10 monclient(hunting): dump: 2024-03-25T18:28:16.424 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:28:16.424 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:28:16.424 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:16.424 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:16.424 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:16.424 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:28:16.424 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:28:16.424 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:28:16.424 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:28:16.424 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:28:16.430 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.429+0000 7f0c2bfff640 1 -- 172.21.15.155:0/3523729997 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2347317324 0 0) 0x7f0c441433d0 con 0x7f0c440b4d60 2024-03-25T18:28:16.430 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.429+0000 7f0c2bfff640 1 -- 172.21.15.155:0/3523729997 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0c30004eb0 con 0x7f0c440b4d60 2024-03-25T18:28:16.430 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.429+0000 7f0c2bfff640 1 -- 172.21.15.155:0/3523729997 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f0c38001ae0 con 0x7f0c440b4d60 2024-03-25T18:28:16.430 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.429+0000 7f0c2bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:16.430 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.429+0000 7f0c2bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:16.430 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.429+0000 7f0c2bfff640 10 monclient(hunting): dump: 2024-03-25T18:28:16.430 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:28:16.430 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:28:16.430 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:16.430 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:16.430 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:16.431 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:28:16.431 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:28:16.431 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:28:16.431 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:28:16.431 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:28:16.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.431+0000 7f0c2bfff640 1 -- 172.21.15.155:0/3523729997 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 596850421 0 0) 0x7f0c44143560 con 0x7f0c4414be90 2024-03-25T18:28:16.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.431+0000 7f0c2bfff640 1 -- 172.21.15.155:0/3523729997 >> v1:172.21.15.167:6789/0 conn(0x7f0c440b1730 legacy=0x7f0c441d79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:16.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.431+0000 7f0c2bfff640 1 -- 172.21.15.155:0/3523729997 >> v1:172.21.15.155:6789/0 conn(0x7f0c440b4d60 legacy=0x7f0c441d80b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:16.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.431+0000 7f0c2bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:28:16.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.431+0000 7f0c2bfff640 1 monclient: found mon.c 2024-03-25T18:28:16.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.431+0000 7f0c2bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:16.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.431+0000 7f0c2bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:28:16.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.431+0000 7f0c2bfff640 1 -- 172.21.15.155:0/3523729997 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c441436f0 con 0x7f0c4414be90 2024-03-25T18:28:16.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.431+0000 7f0c2bfff640 10 monclient: _finish_auth 0 2024-03-25T18:28:16.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.431+0000 7f0c2bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:28:16.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.431+0000 7f0c2bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:16.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.431+0000 7f0c49fc8640 5 monclient: authenticate success, global_id 4385 2024-03-25T18:28:16.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.431+0000 7f0c49fc8640 10 monclient: _renew_subs 2024-03-25T18:28:16.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.431+0000 7f0c49fc8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:28:16.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.431+0000 7f0c49fc8640 1 -- 172.21.15.155:0/3523729997 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c441429a0 con 0x7f0c4414be90 2024-03-25T18:28:16.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.431+0000 7f0c49fc8640 10 client.4385.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:16.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.431+0000 7f0c49fc8640 10 monclient: _renew_subs 2024-03-25T18:28:16.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.431+0000 7f0c49fc8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:28:16.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.431+0000 7f0c49fc8640 1 -- 172.21.15.155:0/3523729997 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c44140560 con 0x7f0c4414be90 2024-03-25T18:28:16.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.431+0000 7f0c49fc8640 1 librados: init done 2024-03-25T18:28:16.434 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.432+0000 7f0c49fc8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:16.434 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.432+0000 7f0c49fc8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:16.434 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.432+0000 7f0c49fc8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:28:16.434 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.432+0000 7f0c49fc8640 1 -- 172.21.15.155:0/3523729997 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c44142ca0 con 0x7f0c4414be90 2024-03-25T18:28:16.750 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.748+0000 7f0c2bfff640 1 -- 172.21.15.155:0/3523729997 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0c3c002060 con 0x7f0c4414be90 2024-03-25T18:28:16.750 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.748+0000 7f0c2bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:16.750 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.748+0000 7f0c2bfff640 1 -- 172.21.15.155:0/3523729997 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f0c3c003b30 con 0x7f0c4414be90 2024-03-25T18:28:16.750 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.748+0000 7f0c2bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:16.750 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.748+0000 7f0c2bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:16.750 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.748+0000 7f0c2bfff640 10 monclient: dump: 2024-03-25T18:28:16.750 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:28:16.750 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:28:16.750 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:16.750 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:16.750 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:16.751 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:28:16.751 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:28:16.751 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:28:16.751 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:28:16.751 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:28:16.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.748+0000 7f0c41ffb640 4 set_mon_vals no callback set 2024-03-25T18:28:16.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.749+0000 7f0c2bfff640 1 -- 172.21.15.155:0/3523729997 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7f0c3c019b40 con 0x7f0c4414be90 2024-03-25T18:28:16.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.750+0000 7f0c437fe640 10 monclient: get_auth_request con 0x7f0c3006e110 auth_method 0 2024-03-25T18:28:16.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.750+0000 7f0c2bfff640 10 client.4385.objecter ms_handle_connect 0x7f0c3006e110 2024-03-25T18:28:16.758 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.757+0000 7f0c2bfff640 1 -- 172.21.15.155:0/3523729997 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7f0c3c0537f0 con 0x7f0c4414be90 2024-03-25T18:28:16.758 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.757+0000 7f0c2bfff640 10 client.4385.objecter ms_dispatch 0x7f0c44000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:16.758 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.757+0000 7f0c2bfff640 3 client.4385.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:16.758 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.757+0000 7f0c2bfff640 3 client.4385.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:16.758 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.757+0000 7f0c2bfff640 20 client.4385.objecter dump_active .. 0 homeless 2024-03-25T18:28:16.758 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.757+0000 7f0c2bfff640 1 -- 172.21.15.155:0/3523729997 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0c3c04f130 con 0x7f0c4414be90 2024-03-25T18:28:16.759 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.757+0000 7f0c2bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:16.759 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:16.757+0000 7f0c2bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:17.011 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:17.010+0000 7f0c49fc8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:17.011 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:17.010+0000 7f0c49fc8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:17.011 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:17.010+0000 7f0c49fc8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:28:17.011 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:17.010+0000 7f0c49fc8640 1 -- 172.21.15.155:0/3523729997 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0c440a97a0 con 0x7f0c4414be90 2024-03-25T18:28:17.045 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:17.044+0000 7f0c2bfff640 1 -- 172.21.15.155:0/3523729997 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7f0c3c051230 con 0x7f0c4414be90 2024-03-25T18:28:17.046 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:17.044+0000 7f0c2bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:17.046 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:17.044+0000 7f0c2bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:17.046 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:28:17.046 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:17.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:17.047+0000 7f0c49fc8640 10 librados: watch_flush enter 2024-03-25T18:28:17.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:17.047+0000 7f0c49fc8640 10 librados: watch_flush exit 2024-03-25T18:28:17.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:17.047+0000 7f0c49fc8640 20 client.4385.objecter shutdown clearing up homeless session... 2024-03-25T18:28:17.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:17.047+0000 7f0c49fc8640 10 client.4385.objecter successfully canceled tick 2024-03-25T18:28:17.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:17.047+0000 7f0c49fc8640 1 -- 172.21.15.155:0/3523729997 >> v1:172.21.15.167:6800/2919070503 conn(0x7f0c3006e110 legacy=0x7f0c300705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:17.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:17.047+0000 7f0c49fc8640 10 monclient: shutdown 2024-03-25T18:28:17.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:17.047+0000 7f0c49fc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:17.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:17.047+0000 7f0c49fc8640 1 -- 172.21.15.155:0/3523729997 >> v1:172.21.15.155:6790/0 conn(0x7f0c4414be90 legacy=0x7f0c441e2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:17.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:17.047+0000 7f0c49fc8640 1 -- 172.21.15.155:0/3523729997 shutdown_connections 2024-03-25T18:28:17.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:17.047+0000 7f0c49fc8640 1 -- 172.21.15.155:0/3523729997 >> 172.21.15.155:0/3523729997 conn(0x7f0c440617a0 msgr2=0x7f0c4413f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:17.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:17.047+0000 7f0c49fc8640 1 -- 172.21.15.155:0/3523729997 shutdown_connections 2024-03-25T18:28:17.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:17.047+0000 7f0c49fc8640 1 -- 172.21.15.155:0/3523729997 wait complete. 2024-03-25T18:28:17.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:17.047+0000 7f0c49fc8640 1 librados: shutdown 2024-03-25T18:28:17.062 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:21.017 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:03:54.622 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:21.025 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:03:54.629 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:21.066 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:03:54.645 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:21.067 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:03:54.672 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:21.068 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:03:54.673 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:21.070 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:03:54.649 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:21.083 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:03:54.688 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:21.105 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:03:54.683 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:21.124 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:03:54.729 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:23.869 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:23.987 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:23.986+0000 7f00b197a640 10 monclient: get_monmap_and_config 2024-03-25T18:28:23.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:23.986+0000 7f00b197a640 10 monclient: build_initial_monmap 2024-03-25T18:28:23.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:23.986+0000 7f00b197a640 10 monclient: monmap: 2024-03-25T18:28:23.988 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:28:23.988 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:28:23.988 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:28:23.987732+0000 2024-03-25T18:28:23.988 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:28:23.987732+0000 2024-03-25T18:28:23.988 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:23.988 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:28:23.988 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:28:23.988 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:28:23.988 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:28:23.988 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:28:23.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:23.987+0000 7f00b197a640 1 Processor -- start 2024-03-25T18:28:23.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:23.987+0000 7f00b197a640 1 -- start start 2024-03-25T18:28:23.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:23.987+0000 7f00b197a640 10 monclient: init 2024-03-25T18:28:23.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:23.988+0000 7f00b197a640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:23.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:23.988+0000 7f00b197a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:28:23.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:23.988+0000 7f00b197a640 10 monclient(hunting): picked mon.noname-c con 0x7f00ac0a98f0 addr v1:172.21.15.167:6789/0 2024-03-25T18:28:23.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:23.988+0000 7f00b197a640 10 monclient(hunting): picked mon.noname-a con 0x7f00ac13f710 addr v1:172.21.15.155:6789/0 2024-03-25T18:28:23.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:23.988+0000 7f00b197a640 10 monclient(hunting): picked mon.noname-b con 0x7f00ac1430c0 addr v1:172.21.15.155:6790/0 2024-03-25T18:28:23.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:23.988+0000 7f00b197a640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00ac142520 con 0x7f00ac13f710 2024-03-25T18:28:23.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:23.988+0000 7f00b197a640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00ac0b28a0 con 0x7f00ac0a98f0 2024-03-25T18:28:23.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:23.988+0000 7f00b197a640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00ac0b2a30 con 0x7f00ac1430c0 2024-03-25T18:28:23.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:23.988+0000 7f00b197a640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:23.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:23.988+0000 7f00b197a640 10 monclient(hunting): authenticate will time out at 7750.298183s 2024-03-25T18:28:23.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:23.997+0000 7f00aa7fc640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f00ac13f710 0x7f00ac1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:50876/0 (socket says 172.21.15.155:50876) 2024-03-25T18:28:23.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:23.997+0000 7f00aa7fc640 1 -- 172.21.15.155:0/2876017187 learned_addr learned my addr 172.21.15.155:0/2876017187 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:28:24.004 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.003+0000 7f00a9ffb640 1 -- 172.21.15.155:0/2876017187 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1753857887 0 0) 0x7f00ac142520 con 0x7f00ac13f710 2024-03-25T18:28:24.004 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.003+0000 7f00a9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:28:24.005 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.003+0000 7f00a9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:24.005 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.003+0000 7f00a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:24.005 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.003+0000 7f00a9ffb640 10 monclient(hunting): my global_id is 4383 2024-03-25T18:28:24.005 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.003+0000 7f00a9ffb640 1 -- 172.21.15.155:0/2876017187 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0094003520 con 0x7f00ac13f710 2024-03-25T18:28:24.015 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.014+0000 7f00a9ffb640 1 -- 172.21.15.155:0/2876017187 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3061708231 0 0) 0x7f0094003520 con 0x7f00ac13f710 2024-03-25T18:28:24.015 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.014+0000 7f00a9ffb640 1 -- 172.21.15.155:0/2876017187 >> v1:172.21.15.155:6790/0 conn(0x7f00ac1430c0 legacy=0x7f00ac14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:24.015 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.014+0000 7f00a9ffb640 1 -- 172.21.15.155:0/2876017187 >> v1:172.21.15.167:6789/0 conn(0x7f00ac0a98f0 legacy=0x7f00ac0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:24.015 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.014+0000 7f00a9ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:28:24.015 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.014+0000 7f00a9ffb640 1 monclient: found mon.noname-a 2024-03-25T18:28:24.016 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.014+0000 7f00a9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:24.016 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.014+0000 7f00a9ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-25T18:28:24.016 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.014+0000 7f00a9ffb640 1 -- 172.21.15.155:0/2876017187 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00ac0b44c0 con 0x7f00ac13f710 2024-03-25T18:28:24.016 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.014+0000 7f00a9ffb640 10 monclient: _finish_auth 0 2024-03-25T18:28:24.016 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.014+0000 7f00a9ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:28:24.016 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.014+0000 7f00a9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:24.016 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.014+0000 7f00a9ffb640 1 -- 172.21.15.155:0/2876017187 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f00a0001690 con 0x7f00ac13f710 2024-03-25T18:28:24.016 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.014+0000 7f00a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:24.016 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.014+0000 7f00a9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:28:24.016 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.014+0000 7f00a9ffb640 10 monclient: dump: 2024-03-25T18:28:24.016 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:28:24.016 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:28:24.016 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:24.016 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:24.017 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:24.017 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:28:24.017 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:28:24.017 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:28:24.017 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:28:24.017 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:28:24.017 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.014+0000 7f00b197a640 5 monclient: authenticate success, global_id 4383 2024-03-25T18:28:24.017 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.014+0000 7f00b197a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:24.029 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.028+0000 7f00a9ffb640 1 -- 172.21.15.155:0/2876017187 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f00a00017f0 con 0x7f00ac13f710 2024-03-25T18:28:24.029 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.028+0000 7f00a9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:24.029 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.028+0000 7f00a9ffb640 1 -- 172.21.15.155:0/2876017187 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f00a0001c00 con 0x7f00ac13f710 2024-03-25T18:28:24.029 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.028+0000 7f00b197a640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:24.029 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.028+0000 7f00b197a640 4 set_mon_vals no callback set 2024-03-25T18:28:24.030 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.028+0000 7f00a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:24.030 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.028+0000 7f00a9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:24.030 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.028+0000 7f00a9ffb640 10 monclient: dump: 2024-03-25T18:28:24.030 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:28:24.030 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:28:24.030 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:24.030 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:24.030 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:24.030 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:28:24.030 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:28:24.030 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:28:24.030 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:28:24.030 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:28:24.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.029+0000 7f00b197a640 10 monclient: shutdown 2024-03-25T18:28:24.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.029+0000 7f00b197a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:24.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.029+0000 7f00b197a640 1 -- 172.21.15.155:0/2876017187 >> v1:172.21.15.155:6789/0 conn(0x7f00ac13f710 legacy=0x7f00ac1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:24.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.029+0000 7f00b197a640 1 -- 172.21.15.155:0/2876017187 shutdown_connections 2024-03-25T18:28:24.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.029+0000 7f00b197a640 1 -- 172.21.15.155:0/2876017187 >> 172.21.15.155:0/2876017187 conn(0x7f00ac0b7ea0 msgr2=0x7f00ac0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:24.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.029+0000 7f00b197a640 1 -- 172.21.15.155:0/2876017187 shutdown_connections 2024-03-25T18:28:24.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.029+0000 7f00b197a640 1 -- 172.21.15.155:0/2876017187 wait complete. 2024-03-25T18:28:24.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.030+0000 7f00b197a640 10 monclient: build_initial_monmap 2024-03-25T18:28:24.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.030+0000 7f00b197a640 10 monclient: monmap: 2024-03-25T18:28:24.034 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:28:24.034 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:24.034 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:28:24.034 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:28:24.034 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:24.034 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:28:24.034 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:28:24.035 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:28:24.035 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:28:24.035 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:28:24.035 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.030+0000 7f00b197a640 1 librados: starting msgr at 2024-03-25T18:28:24.035 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.030+0000 7f00b197a640 1 librados: starting objecter 2024-03-25T18:28:24.035 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.031+0000 7f00b197a640 1 Processor -- start 2024-03-25T18:28:24.035 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.031+0000 7f00b197a640 1 -- start start 2024-03-25T18:28:24.035 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.031+0000 7f00b197a640 1 librados: setting wanted keys 2024-03-25T18:28:24.035 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.031+0000 7f00b197a640 1 librados: calling monclient init 2024-03-25T18:28:24.035 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.031+0000 7f00b197a640 10 monclient: init 2024-03-25T18:28:24.036 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.031+0000 7f00b197a640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:24.036 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.031+0000 7f00b197a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:28:24.036 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.031+0000 7f00b197a640 10 monclient(hunting): picked mon.noname-c con 0x7f00ac0a98f0 addr v1:172.21.15.155:6790/0 2024-03-25T18:28:24.036 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.031+0000 7f00b197a640 10 monclient(hunting): picked mon.noname-b con 0x7f00ac13f710 addr v1:172.21.15.167:6789/0 2024-03-25T18:28:24.037 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.031+0000 7f00b197a640 10 monclient(hunting): picked mon.noname-a con 0x7f00ac1430c0 addr v1:172.21.15.155:6789/0 2024-03-25T18:28:24.037 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.031+0000 7f00b197a640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00ac138980 con 0x7f00ac1430c0 2024-03-25T18:28:24.037 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.031+0000 7f00b197a640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00ac138640 con 0x7f00ac13f710 2024-03-25T18:28:24.037 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.031+0000 7f00b197a640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00ac0b1700 con 0x7f00ac0a98f0 2024-03-25T18:28:24.037 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.031+0000 7f00b197a640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:24.037 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.031+0000 7f00b197a640 10 monclient(hunting): authenticate will time out at 7750.341597s 2024-03-25T18:28:24.043 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.042+0000 7f00aaffd640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f00ac0a98f0 0x7f00ac137960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:55826/0 (socket says 172.21.15.155:55826) 2024-03-25T18:28:24.043 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.042+0000 7f00aaffd640 1 -- 172.21.15.155:0/1954058906 learned_addr learned my addr 172.21.15.155:0/1954058906 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:28:24.043 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.043+0000 7f0093fff640 10 client.?.objecter ms_handle_connect 0x7f00ac1430c0 2024-03-25T18:28:24.044 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.043+0000 7f0093fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:24.045 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.044+0000 7f0093fff640 10 client.?.objecter ms_handle_connect 0x7f00ac0a98f0 2024-03-25T18:28:24.046 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.044+0000 7f0093fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:24.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.049+0000 7f0093fff640 10 client.?.objecter ms_handle_connect 0x7f00ac13f710 2024-03-25T18:28:24.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.049+0000 7f0093fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:24.054 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.053+0000 7f0093fff640 1 -- 172.21.15.155:0/1954058906 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 790261639 0 0) 0x7f00ac138980 con 0x7f00ac1430c0 2024-03-25T18:28:24.055 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.053+0000 7f0093fff640 10 monclient(hunting): _negotiate 2024-03-25T18:28:24.055 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.053+0000 7f0093fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:24.055 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.053+0000 7f0093fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:24.055 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.053+0000 7f0093fff640 10 monclient(hunting): my global_id is 4386 2024-03-25T18:28:24.055 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.054+0000 7f0093fff640 1 -- 172.21.15.155:0/1954058906 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0098004eb0 con 0x7f00ac1430c0 2024-03-25T18:28:24.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.060+0000 7f0093fff640 1 -- 172.21.15.155:0/1954058906 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3930320583 0 0) 0x7f00ac138640 con 0x7f00ac13f710 2024-03-25T18:28:24.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.060+0000 7f0093fff640 10 monclient(hunting): _negotiate 2024-03-25T18:28:24.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.060+0000 7f0093fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:24.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.060+0000 7f0093fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:24.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.060+0000 7f0093fff640 10 monclient(hunting): my global_id is 4381 2024-03-25T18:28:24.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.060+0000 7f0093fff640 1 -- 172.21.15.155:0/1954058906 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00ac138980 con 0x7f00ac13f710 2024-03-25T18:28:24.064 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.063+0000 7f0093fff640 1 -- 172.21.15.155:0/1954058906 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4060191783 0 0) 0x7f0098004eb0 con 0x7f00ac1430c0 2024-03-25T18:28:24.064 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.063+0000 7f0093fff640 1 -- 172.21.15.155:0/1954058906 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f00ac138640 con 0x7f00ac1430c0 2024-03-25T18:28:24.064 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.063+0000 7f0093fff640 1 -- 172.21.15.155:0/1954058906 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f00a4001b90 con 0x7f00ac1430c0 2024-03-25T18:28:24.065 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.063+0000 7f0093fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:24.065 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.063+0000 7f0093fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:28:24.065 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.063+0000 7f0093fff640 10 monclient(hunting): dump: 2024-03-25T18:28:24.065 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:28:24.065 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:28:24.065 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:24.065 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:24.065 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:24.065 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:28:24.066 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:28:24.066 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:28:24.066 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:28:24.066 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:28:24.066 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.063+0000 7f0093fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:24.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.071+0000 7f0093fff640 1 -- 172.21.15.155:0/1954058906 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1830379838 0 0) 0x7f00ac0b1700 con 0x7f00ac0a98f0 2024-03-25T18:28:24.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.071+0000 7f0093fff640 10 monclient(hunting): _negotiate 2024-03-25T18:28:24.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.071+0000 7f0093fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:24.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.071+0000 7f0093fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:24.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.071+0000 7f0093fff640 10 monclient(hunting): my global_id is 4391 2024-03-25T18:28:24.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.071+0000 7f0093fff640 1 -- 172.21.15.155:0/1954058906 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0098004eb0 con 0x7f00ac0a98f0 2024-03-25T18:28:24.074 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.073+0000 7f0093fff640 1 -- 172.21.15.155:0/1954058906 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 897983847 0 0) 0x7f00ac138980 con 0x7f00ac13f710 2024-03-25T18:28:24.074 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.073+0000 7f0093fff640 1 -- 172.21.15.155:0/1954058906 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f00ac0b1700 con 0x7f00ac13f710 2024-03-25T18:28:24.074 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.073+0000 7f0093fff640 1 -- 172.21.15.155:0/1954058906 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f00a0001b00 con 0x7f00ac13f710 2024-03-25T18:28:24.074 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.073+0000 7f0093fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:24.074 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.073+0000 7f0093fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:24.074 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.073+0000 7f0093fff640 10 monclient(hunting): dump: 2024-03-25T18:28:24.075 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:28:24.075 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:28:24.075 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:24.075 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:24.075 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:24.075 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:28:24.075 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:28:24.075 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:28:24.075 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:28:24.076 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:28:24.079 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.076+0000 7f0093fff640 1 -- 172.21.15.155:0/1954058906 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2344074473 0 0) 0x7f00ac138640 con 0x7f00ac1430c0 2024-03-25T18:28:24.079 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.077+0000 7f0093fff640 1 -- 172.21.15.155:0/1954058906 >> v1:172.21.15.155:6790/0 conn(0x7f00ac0a98f0 legacy=0x7f00ac137960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:24.079 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.077+0000 7f0093fff640 1 -- 172.21.15.155:0/1954058906 >> v1:172.21.15.167:6789/0 conn(0x7f00ac13f710 legacy=0x7f00ac1da8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:24.079 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.077+0000 7f0093fff640 10 monclient: _finish_hunting 0 2024-03-25T18:28:24.079 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.077+0000 7f0093fff640 1 monclient: found mon.a 2024-03-25T18:28:24.079 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.077+0000 7f0093fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:24.079 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.077+0000 7f0093fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:28:24.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.077+0000 7f0093fff640 1 -- 172.21.15.155:0/1954058906 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00ac0b1290 con 0x7f00ac1430c0 2024-03-25T18:28:24.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.077+0000 7f0093fff640 10 monclient: _finish_auth 0 2024-03-25T18:28:24.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.077+0000 7f0093fff640 10 monclient: _check_auth_tickets 2024-03-25T18:28:24.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.077+0000 7f0093fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:24.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.077+0000 7f00b197a640 5 monclient: authenticate success, global_id 4386 2024-03-25T18:28:24.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.077+0000 7f00b197a640 10 monclient: _renew_subs 2024-03-25T18:28:24.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.077+0000 7f00b197a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:28:24.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.077+0000 7f00b197a640 1 -- 172.21.15.155:0/1954058906 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00ac1e2eb0 con 0x7f00ac1430c0 2024-03-25T18:28:24.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.077+0000 7f00b197a640 10 client.4386.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:24.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.077+0000 7f00b197a640 10 monclient: _renew_subs 2024-03-25T18:28:24.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.077+0000 7f00b197a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:28:24.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.077+0000 7f00b197a640 1 -- 172.21.15.155:0/1954058906 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f00ac1e3220 con 0x7f00ac1430c0 2024-03-25T18:28:24.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.077+0000 7f00b197a640 1 librados: init done 2024-03-25T18:28:24.081 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.078+0000 7f00b197a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:24.081 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.078+0000 7f00b197a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:24.081 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.078+0000 7f00b197a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:28:24.081 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.078+0000 7f00b197a640 1 -- 172.21.15.155:0/1954058906 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00ac0abb30 con 0x7f00ac1430c0 2024-03-25T18:28:24.405 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.403+0000 7f0093fff640 1 -- 172.21.15.155:0/1954058906 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f00a4002080 con 0x7f00ac1430c0 2024-03-25T18:28:24.405 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.403+0000 7f0093fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:24.405 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.403+0000 7f0093fff640 1 -- 172.21.15.155:0/1954058906 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f00a4003b10 con 0x7f00ac1430c0 2024-03-25T18:28:24.405 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.403+0000 7f0093fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:24.405 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.403+0000 7f0093fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:24.405 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.403+0000 7f0093fff640 10 monclient: dump: 2024-03-25T18:28:24.405 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:28:24.405 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:28:24.405 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:24.405 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:24.405 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:24.405 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:28:24.406 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:28:24.406 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:28:24.406 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:28:24.406 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:28:24.406 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.403+0000 7f00a97fa640 4 set_mon_vals no callback set 2024-03-25T18:28:24.406 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.404+0000 7f0093fff640 1 -- 172.21.15.155:0/1954058906 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7f00a4003d30 con 0x7f00ac1430c0 2024-03-25T18:28:24.407 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.405+0000 7f00aaffd640 10 monclient: get_auth_request con 0x7f009806df70 auth_method 0 2024-03-25T18:28:24.407 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.405+0000 7f0093fff640 10 client.4386.objecter ms_handle_connect 0x7f009806df70 2024-03-25T18:28:24.411 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.410+0000 7f0093fff640 1 -- 172.21.15.155:0/1954058906 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7f00a4053860 con 0x7f00ac1430c0 2024-03-25T18:28:24.412 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.410+0000 7f0093fff640 10 client.4386.objecter ms_dispatch 0x7f00ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:24.412 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.410+0000 7f0093fff640 3 client.4386.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:24.412 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.410+0000 7f0093fff640 3 client.4386.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:24.412 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.411+0000 7f0093fff640 20 client.4386.objecter dump_active .. 0 homeless 2024-03-25T18:28:24.412 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.411+0000 7f0093fff640 1 -- 172.21.15.155:0/1954058906 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f00a404f370 con 0x7f00ac1430c0 2024-03-25T18:28:24.412 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.411+0000 7f0093fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:24.412 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.411+0000 7f0093fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:24.620 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.619+0000 7f00b197a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:24.620 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.619+0000 7f00b197a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:24.620 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.619+0000 7f00b197a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:28:24.620 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.619+0000 7f00b197a640 1 -- 172.21.15.155:0/1954058906 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f00ac0627b0 con 0x7f00ac1430c0 2024-03-25T18:28:24.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.644+0000 7f0093fff640 1 -- 172.21.15.155:0/1954058906 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7f00a40512a0 con 0x7f00ac1430c0 2024-03-25T18:28:24.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.644+0000 7f0093fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:24.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.644+0000 7f0093fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:24.645 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:28:24.646 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:24.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.646+0000 7f00b197a640 10 librados: watch_flush enter 2024-03-25T18:28:24.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.646+0000 7f00b197a640 10 librados: watch_flush exit 2024-03-25T18:28:24.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.646+0000 7f00b197a640 20 client.4386.objecter shutdown clearing up homeless session... 2024-03-25T18:28:24.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.646+0000 7f00b197a640 10 client.4386.objecter successfully canceled tick 2024-03-25T18:28:24.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.646+0000 7f00b197a640 1 -- 172.21.15.155:0/1954058906 >> v1:172.21.15.167:6800/2919070503 conn(0x7f009806df70 legacy=0x7f0098070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:24.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.646+0000 7f00b197a640 10 monclient: shutdown 2024-03-25T18:28:24.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.646+0000 7f00b197a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:24.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.646+0000 7f00b197a640 1 -- 172.21.15.155:0/1954058906 >> v1:172.21.15.155:6789/0 conn(0x7f00ac1430c0 legacy=0x7f00ac1db000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:24.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.646+0000 7f00b197a640 1 -- 172.21.15.155:0/1954058906 shutdown_connections 2024-03-25T18:28:24.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.646+0000 7f00b197a640 1 -- 172.21.15.155:0/1954058906 >> 172.21.15.155:0/1954058906 conn(0x7f00ac0b7ea0 msgr2=0x7f00ac074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:24.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.646+0000 7f00b197a640 1 -- 172.21.15.155:0/1954058906 shutdown_connections 2024-03-25T18:28:24.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.646+0000 7f00b197a640 1 -- 172.21.15.155:0/1954058906 wait complete. 2024-03-25T18:28:24.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:24.646+0000 7f00b197a640 1 librados: shutdown 2024-03-25T18:28:24.660 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:31.468 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:31.585 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.584+0000 7fa575e86640 10 monclient: get_monmap_and_config 2024-03-25T18:28:31.585 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.584+0000 7fa575e86640 10 monclient: build_initial_monmap 2024-03-25T18:28:31.585 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.584+0000 7fa575e86640 10 monclient: monmap: 2024-03-25T18:28:31.586 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:28:31.586 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:28:31.586 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:28:31.585445+0000 2024-03-25T18:28:31.586 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:28:31.585445+0000 2024-03-25T18:28:31.586 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:31.586 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:28:31.586 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:28:31.586 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:28:31.586 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:28:31.586 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:28:31.587 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.585+0000 7fa575e86640 1 Processor -- start 2024-03-25T18:28:31.588 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.585+0000 7fa575e86640 1 -- start start 2024-03-25T18:28:31.588 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.585+0000 7fa575e86640 10 monclient: init 2024-03-25T18:28:31.588 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.586+0000 7fa575e86640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:31.588 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.586+0000 7fa575e86640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:28:31.588 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.586+0000 7fa575e86640 10 monclient(hunting): picked mon.noname-b con 0x7fa5700a98f0 addr v1:172.21.15.155:6790/0 2024-03-25T18:28:31.588 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.586+0000 7fa575e86640 10 monclient(hunting): picked mon.noname-c con 0x7fa57013f710 addr v1:172.21.15.167:6789/0 2024-03-25T18:28:31.588 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.586+0000 7fa575e86640 10 monclient(hunting): picked mon.noname-a con 0x7fa5701430c0 addr v1:172.21.15.155:6789/0 2024-03-25T18:28:31.588 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.586+0000 7fa575e86640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa570142520 con 0x7fa5701430c0 2024-03-25T18:28:31.588 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.586+0000 7fa575e86640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5700b28a0 con 0x7fa57013f710 2024-03-25T18:28:31.588 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.586+0000 7fa575e86640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5700b2a30 con 0x7fa5700a98f0 2024-03-25T18:28:31.588 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.586+0000 7fa575e86640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:31.588 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.586+0000 7fa575e86640 10 monclient(hunting): authenticate will time out at 7757.896137s 2024-03-25T18:28:31.595 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.594+0000 7fa56effd640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7fa57013f710 0x7fa5701429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:58722/0 (socket says 172.21.15.155:58722) 2024-03-25T18:28:31.595 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.594+0000 7fa56effd640 1 -- 172.21.15.155:0/3988389124 learned_addr learned my addr 172.21.15.155:0/3988389124 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:28:31.608 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.607+0000 7fa56e7fc640 1 -- 172.21.15.155:0/3988389124 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 195568148 0 0) 0x7fa5700b28a0 con 0x7fa57013f710 2024-03-25T18:28:31.608 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.607+0000 7fa56e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:31.608 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.607+0000 7fa56e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:31.608 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.607+0000 7fa56e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:31.608 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.607+0000 7fa56e7fc640 10 monclient(hunting): my global_id is 4384 2024-03-25T18:28:31.608 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.607+0000 7fa56e7fc640 1 -- 172.21.15.155:0/3988389124 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa558003520 con 0x7fa57013f710 2024-03-25T18:28:31.612 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.612+0000 7fa56e7fc640 1 -- 172.21.15.155:0/3988389124 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2507944102 0 0) 0x7fa570142520 con 0x7fa5701430c0 2024-03-25T18:28:31.612 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.612+0000 7fa56e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:31.613 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.612+0000 7fa56e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:31.613 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.612+0000 7fa56e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:31.613 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.612+0000 7fa56e7fc640 10 monclient(hunting): my global_id is 4389 2024-03-25T18:28:31.613 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.612+0000 7fa56e7fc640 1 -- 172.21.15.155:0/3988389124 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5700b28a0 con 0x7fa5701430c0 2024-03-25T18:28:31.613 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.612+0000 7fa56e7fc640 1 -- 172.21.15.155:0/3988389124 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3095832834 0 0) 0x7fa5700b2a30 con 0x7fa5700a98f0 2024-03-25T18:28:31.613 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.612+0000 7fa56e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:31.614 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.612+0000 7fa56e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:31.614 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.612+0000 7fa56e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:31.614 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.612+0000 7fa56e7fc640 10 monclient(hunting): my global_id is 4394 2024-03-25T18:28:31.614 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.612+0000 7fa56e7fc640 1 -- 172.21.15.155:0/3988389124 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa570142520 con 0x7fa5700a98f0 2024-03-25T18:28:31.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.620+0000 7fa56e7fc640 1 -- 172.21.15.155:0/3988389124 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2811033671 0 0) 0x7fa558003520 con 0x7fa57013f710 2024-03-25T18:28:31.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.620+0000 7fa56e7fc640 1 -- 172.21.15.155:0/3988389124 >> v1:172.21.15.155:6790/0 conn(0x7fa5700a98f0 legacy=0x7fa5700a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:31.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.620+0000 7fa56e7fc640 1 -- 172.21.15.155:0/3988389124 >> v1:172.21.15.155:6789/0 conn(0x7fa5701430c0 legacy=0x7fa57014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:31.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.620+0000 7fa56e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:28:31.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.620+0000 7fa56e7fc640 1 monclient: found mon.noname-c 2024-03-25T18:28:31.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.620+0000 7fa56e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:31.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.620+0000 7fa56e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.167:6789/0 2024-03-25T18:28:31.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.621+0000 7fa56e7fc640 1 -- 172.21.15.155:0/3988389124 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5700b44c0 con 0x7fa57013f710 2024-03-25T18:28:31.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.621+0000 7fa56e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:28:31.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.621+0000 7fa56e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:28:31.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.621+0000 7fa56e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:31.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.621+0000 7fa56e7fc640 1 -- 172.21.15.155:0/3988389124 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fa564001690 con 0x7fa57013f710 2024-03-25T18:28:31.623 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.621+0000 7fa56e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:31.623 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.621+0000 7fa56e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:28:31.623 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.621+0000 7fa56e7fc640 10 monclient: dump: 2024-03-25T18:28:31.623 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:28:31.623 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:28:31.623 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:31.623 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:31.623 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:31.623 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:28:31.623 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:28:31.623 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:28:31.623 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:28:31.623 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:28:31.623 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.621+0000 7fa575e86640 5 monclient: authenticate success, global_id 4384 2024-03-25T18:28:31.624 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.621+0000 7fa575e86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:31.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.645+0000 7fa56e7fc640 1 -- 172.21.15.155:0/3988389124 <== mon.1 v1:172.21.15.167:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa5640017f0 con 0x7fa57013f710 2024-03-25T18:28:31.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.645+0000 7fa56e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:31.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.645+0000 7fa56e7fc640 1 -- 172.21.15.155:0/3988389124 <== mon.1 v1:172.21.15.167:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fa564001c00 con 0x7fa57013f710 2024-03-25T18:28:31.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.645+0000 7fa56e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:31.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.645+0000 7fa56e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:31.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.645+0000 7fa56e7fc640 10 monclient: dump: 2024-03-25T18:28:31.647 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:28:31.647 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:28:31.647 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:31.647 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:31.647 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:31.647 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:28:31.647 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:28:31.647 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:28:31.647 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:28:31.647 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:28:31.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.645+0000 7fa575e86640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:31.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.645+0000 7fa575e86640 4 set_mon_vals no callback set 2024-03-25T18:28:31.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.646+0000 7fa575e86640 10 monclient: shutdown 2024-03-25T18:28:31.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.646+0000 7fa575e86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:31.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.646+0000 7fa575e86640 1 -- 172.21.15.155:0/3988389124 >> v1:172.21.15.167:6789/0 conn(0x7fa57013f710 legacy=0x7fa5701429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:31.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.646+0000 7fa575e86640 1 -- 172.21.15.155:0/3988389124 shutdown_connections 2024-03-25T18:28:31.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.646+0000 7fa575e86640 1 -- 172.21.15.155:0/3988389124 >> 172.21.15.155:0/3988389124 conn(0x7fa5700b7ea0 msgr2=0x7fa5700b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:31.653 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.646+0000 7fa575e86640 1 -- 172.21.15.155:0/3988389124 shutdown_connections 2024-03-25T18:28:31.653 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.646+0000 7fa575e86640 1 -- 172.21.15.155:0/3988389124 wait complete. 2024-03-25T18:28:31.653 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.647+0000 7fa575e86640 10 monclient: build_initial_monmap 2024-03-25T18:28:31.653 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.647+0000 7fa575e86640 10 monclient: monmap: 2024-03-25T18:28:31.653 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:28:31.653 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:31.654 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:28:31.654 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:28:31.654 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:31.654 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:28:31.654 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:28:31.655 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:28:31.655 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:28:31.655 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:28:31.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.647+0000 7fa575e86640 1 librados: starting msgr at 2024-03-25T18:28:31.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.647+0000 7fa575e86640 1 librados: starting objecter 2024-03-25T18:28:31.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.647+0000 7fa575e86640 1 Processor -- start 2024-03-25T18:28:31.656 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.647+0000 7fa575e86640 1 -- start start 2024-03-25T18:28:31.656 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.647+0000 7fa575e86640 1 librados: setting wanted keys 2024-03-25T18:28:31.656 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.647+0000 7fa575e86640 1 librados: calling monclient init 2024-03-25T18:28:31.656 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.647+0000 7fa575e86640 10 monclient: init 2024-03-25T18:28:31.656 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.648+0000 7fa575e86640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:31.656 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.648+0000 7fa575e86640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:28:31.656 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.648+0000 7fa575e86640 10 monclient(hunting): picked mon.noname-b con 0x7fa5700a98f0 addr v1:172.21.15.167:6789/0 2024-03-25T18:28:31.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.648+0000 7fa575e86640 10 monclient(hunting): picked mon.noname-c con 0x7fa57013f710 addr v1:172.21.15.155:6790/0 2024-03-25T18:28:31.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.648+0000 7fa575e86640 10 monclient(hunting): picked mon.noname-a con 0x7fa5701430c0 addr v1:172.21.15.155:6789/0 2024-03-25T18:28:31.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.648+0000 7fa575e86640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5701d77a0 con 0x7fa5701430c0 2024-03-25T18:28:31.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.648+0000 7fa575e86640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa570138e40 con 0x7fa5700a98f0 2024-03-25T18:28:31.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.648+0000 7fa575e86640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5700aba30 con 0x7fa57013f710 2024-03-25T18:28:31.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.648+0000 7fa575e86640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:31.658 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.648+0000 7fa575e86640 10 monclient(hunting): authenticate will time out at 7757.958490s 2024-03-25T18:28:31.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.655+0000 7fa56ffff640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fa5701430c0 0x7fa570134110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:40890/0 (socket says 172.21.15.155:40890) 2024-03-25T18:28:31.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.655+0000 7fa56ffff640 1 -- 172.21.15.155:0/2849094622 learned_addr learned my addr 172.21.15.155:0/2849094622 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:28:31.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.657+0000 7fa557fff640 10 client.?.objecter ms_handle_connect 0x7fa5701430c0 2024-03-25T18:28:31.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.657+0000 7fa557fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:31.663 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.662+0000 7fa557fff640 10 client.?.objecter ms_handle_connect 0x7fa5700a98f0 2024-03-25T18:28:31.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.662+0000 7fa557fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:31.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.664+0000 7fa557fff640 10 client.?.objecter ms_handle_connect 0x7fa57013f710 2024-03-25T18:28:31.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.664+0000 7fa557fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:31.667 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.666+0000 7fa557fff640 1 -- 172.21.15.155:0/2849094622 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3662827694 0 0) 0x7fa5701d77a0 con 0x7fa5701430c0 2024-03-25T18:28:31.667 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.666+0000 7fa557fff640 10 monclient(hunting): _negotiate 2024-03-25T18:28:31.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.666+0000 7fa557fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:31.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.666+0000 7fa557fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:31.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.666+0000 7fa557fff640 10 monclient(hunting): my global_id is 4392 2024-03-25T18:28:31.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.666+0000 7fa557fff640 1 -- 172.21.15.155:0/2849094622 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa55c004eb0 con 0x7fa5701430c0 2024-03-25T18:28:31.670 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.669+0000 7fa557fff640 1 -- 172.21.15.155:0/2849094622 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 59177592 0 0) 0x7fa570138e40 con 0x7fa5700a98f0 2024-03-25T18:28:31.670 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.669+0000 7fa557fff640 10 monclient(hunting): _negotiate 2024-03-25T18:28:31.670 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.669+0000 7fa557fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:31.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.669+0000 7fa557fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:31.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.669+0000 7fa557fff640 10 monclient(hunting): my global_id is 4387 2024-03-25T18:28:31.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.669+0000 7fa557fff640 1 -- 172.21.15.155:0/2849094622 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5701d77a0 con 0x7fa5700a98f0 2024-03-25T18:28:31.675 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.674+0000 7fa557fff640 1 -- 172.21.15.155:0/2849094622 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2006018338 0 0) 0x7fa5700aba30 con 0x7fa57013f710 2024-03-25T18:28:31.675 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.674+0000 7fa557fff640 10 monclient(hunting): _negotiate 2024-03-25T18:28:31.675 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.674+0000 7fa557fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:31.675 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.674+0000 7fa557fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:31.675 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.674+0000 7fa557fff640 10 monclient(hunting): my global_id is 4397 2024-03-25T18:28:31.675 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.674+0000 7fa557fff640 1 -- 172.21.15.155:0/2849094622 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa570138e40 con 0x7fa57013f710 2024-03-25T18:28:31.680 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.679+0000 7fa557fff640 1 -- 172.21.15.155:0/2849094622 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4086360964 0 0) 0x7fa55c004eb0 con 0x7fa5701430c0 2024-03-25T18:28:31.680 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.679+0000 7fa557fff640 1 -- 172.21.15.155:0/2849094622 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5700aba30 con 0x7fa5701430c0 2024-03-25T18:28:31.680 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.679+0000 7fa557fff640 1 -- 172.21.15.155:0/2849094622 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fa568001c90 con 0x7fa5701430c0 2024-03-25T18:28:31.680 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.679+0000 7fa557fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:31.680 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.679+0000 7fa557fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:28:31.680 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.679+0000 7fa557fff640 10 monclient(hunting): dump: 2024-03-25T18:28:31.680 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:28:31.680 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:28:31.680 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:31.681 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:31.681 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:31.681 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:28:31.681 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:28:31.681 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:28:31.681 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:28:31.681 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:28:31.681 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.679+0000 7fa557fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:31.687 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.685+0000 7fa557fff640 1 -- 172.21.15.155:0/2849094622 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3449151316 0 0) 0x7fa5700aba30 con 0x7fa5701430c0 2024-03-25T18:28:31.687 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.685+0000 7fa557fff640 1 -- 172.21.15.155:0/2849094622 >> v1:172.21.15.155:6790/0 conn(0x7fa57013f710 legacy=0x7fa5701d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:31.687 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.685+0000 7fa557fff640 1 -- 172.21.15.155:0/2849094622 >> v1:172.21.15.167:6789/0 conn(0x7fa5700a98f0 legacy=0x7fa570138730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:31.687 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.685+0000 7fa557fff640 10 monclient: _finish_hunting 0 2024-03-25T18:28:31.687 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.685+0000 7fa557fff640 1 monclient: found mon.a 2024-03-25T18:28:31.687 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.685+0000 7fa557fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:31.687 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.685+0000 7fa557fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:28:31.687 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.685+0000 7fa557fff640 1 -- 172.21.15.155:0/2849094622 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5701d8340 con 0x7fa5701430c0 2024-03-25T18:28:31.687 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.685+0000 7fa557fff640 10 monclient: _finish_auth 0 2024-03-25T18:28:31.688 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.685+0000 7fa557fff640 10 monclient: _check_auth_tickets 2024-03-25T18:28:31.688 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.685+0000 7fa557fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:31.688 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.685+0000 7fa575e86640 5 monclient: authenticate success, global_id 4392 2024-03-25T18:28:31.688 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.685+0000 7fa575e86640 10 monclient: _renew_subs 2024-03-25T18:28:31.688 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.685+0000 7fa575e86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:28:31.688 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.685+0000 7fa575e86640 1 -- 172.21.15.155:0/2849094622 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa570134820 con 0x7fa5701430c0 2024-03-25T18:28:31.688 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.686+0000 7fa575e86640 10 client.4392.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:31.688 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.686+0000 7fa575e86640 10 monclient: _renew_subs 2024-03-25T18:28:31.688 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.686+0000 7fa575e86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:28:31.688 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.686+0000 7fa575e86640 1 -- 172.21.15.155:0/2849094622 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5701380c0 con 0x7fa5701430c0 2024-03-25T18:28:31.688 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.686+0000 7fa575e86640 1 librados: init done 2024-03-25T18:28:31.689 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.687+0000 7fa575e86640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:31.689 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.687+0000 7fa575e86640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:31.689 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.687+0000 7fa575e86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:28:31.689 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.687+0000 7fa575e86640 1 -- 172.21.15.155:0/2849094622 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5700627b0 con 0x7fa5701430c0 2024-03-25T18:28:31.997 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.995+0000 7fa557fff640 1 -- 172.21.15.155:0/2849094622 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa568003970 con 0x7fa5701430c0 2024-03-25T18:28:31.997 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.995+0000 7fa557fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:31.997 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.995+0000 7fa557fff640 1 -- 172.21.15.155:0/2849094622 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fa568003d40 con 0x7fa5701430c0 2024-03-25T18:28:31.997 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.995+0000 7fa557fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:31.997 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.995+0000 7fa557fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:31.997 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.995+0000 7fa557fff640 10 monclient: dump: 2024-03-25T18:28:31.997 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:28:31.997 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:28:31.997 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:31.997 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:31.998 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:31.998 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:28:31.998 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:28:31.998 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:28:31.998 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:28:31.998 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:28:31.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.995+0000 7fa56dffb640 4 set_mon_vals no callback set 2024-03-25T18:28:31.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.996+0000 7fa557fff640 1 -- 172.21.15.155:0/2849094622 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7fa568003f60 con 0x7fa5701430c0 2024-03-25T18:28:31.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.997+0000 7fa56f7fe640 10 monclient: get_auth_request con 0x7fa55c06de20 auth_method 0 2024-03-25T18:28:31.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:31.997+0000 7fa557fff640 10 client.4392.objecter ms_handle_connect 0x7fa55c06de20 2024-03-25T18:28:32.004 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:32.002+0000 7fa557fff640 1 -- 172.21.15.155:0/2849094622 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7fa568054b30 con 0x7fa5701430c0 2024-03-25T18:28:32.004 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:32.002+0000 7fa557fff640 10 client.4392.objecter ms_dispatch 0x7fa570000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:32.004 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:32.002+0000 7fa557fff640 3 client.4392.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:32.005 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:32.002+0000 7fa557fff640 3 client.4392.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:32.005 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:32.002+0000 7fa557fff640 20 client.4392.objecter dump_active .. 0 homeless 2024-03-25T18:28:32.005 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:32.002+0000 7fa557fff640 1 -- 172.21.15.155:0/2849094622 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa568050640 con 0x7fa5701430c0 2024-03-25T18:28:32.005 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:32.003+0000 7fa557fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:32.005 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:32.003+0000 7fa557fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:32.258 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:32.256+0000 7fa575e86640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:32.258 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:32.256+0000 7fa575e86640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:32.258 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:32.256+0000 7fa575e86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:28:32.258 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:32.256+0000 7fa575e86640 1 -- 172.21.15.155:0/2849094622 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa57014e8b0 con 0x7fa5701430c0 2024-03-25T18:28:32.320 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:32.319+0000 7fa557fff640 1 -- 172.21.15.155:0/2849094622 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7fa568052570 con 0x7fa5701430c0 2024-03-25T18:28:32.320 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:32.319+0000 7fa557fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:32.320 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:32.319+0000 7fa557fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:32.321 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:28:32.321 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:32.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:32.321+0000 7fa575e86640 10 librados: watch_flush enter 2024-03-25T18:28:32.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:32.321+0000 7fa575e86640 10 librados: watch_flush exit 2024-03-25T18:28:32.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:32.321+0000 7fa575e86640 20 client.4392.objecter shutdown clearing up homeless session... 2024-03-25T18:28:32.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:32.321+0000 7fa575e86640 10 client.4392.objecter successfully canceled tick 2024-03-25T18:28:32.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:32.321+0000 7fa575e86640 1 -- 172.21.15.155:0/2849094622 >> v1:172.21.15.167:6800/2919070503 conn(0x7fa55c06de20 legacy=0x7fa55c0702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:32.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:32.321+0000 7fa575e86640 10 monclient: shutdown 2024-03-25T18:28:32.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:32.321+0000 7fa575e86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:32.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:32.321+0000 7fa575e86640 1 -- 172.21.15.155:0/2849094622 >> v1:172.21.15.155:6789/0 conn(0x7fa5701430c0 legacy=0x7fa570134110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:32.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:32.321+0000 7fa575e86640 1 -- 172.21.15.155:0/2849094622 shutdown_connections 2024-03-25T18:28:32.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:32.321+0000 7fa575e86640 1 -- 172.21.15.155:0/2849094622 >> 172.21.15.155:0/2849094622 conn(0x7fa5700b7ea0 msgr2=0x7fa570074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:32.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:32.322+0000 7fa575e86640 1 -- 172.21.15.155:0/2849094622 shutdown_connections 2024-03-25T18:28:32.324 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:32.322+0000 7fa575e86640 1 -- 172.21.15.155:0/2849094622 wait complete. 2024-03-25T18:28:32.324 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:32.322+0000 7fa575e86640 1 librados: shutdown 2024-03-25T18:28:32.336 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:39.143 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:39.242 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.241+0000 7f901ee1e640 10 monclient: get_monmap_and_config 2024-03-25T18:28:39.242 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.241+0000 7f901ee1e640 10 monclient: build_initial_monmap 2024-03-25T18:28:39.242 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.241+0000 7f901ee1e640 10 monclient: monmap: 2024-03-25T18:28:39.242 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:28:39.242 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:28:39.243 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:28:39.242426+0000 2024-03-25T18:28:39.243 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:28:39.242426+0000 2024-03-25T18:28:39.243 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:39.243 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:28:39.243 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:28:39.243 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:28:39.243 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:28:39.243 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:28:39.244 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.242+0000 7f901ee1e640 1 Processor -- start 2024-03-25T18:28:39.244 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.242+0000 7f901ee1e640 1 -- start start 2024-03-25T18:28:39.244 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.242+0000 7f901ee1e640 10 monclient: init 2024-03-25T18:28:39.245 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.242+0000 7f901ee1e640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:39.245 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.242+0000 7f901ee1e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:28:39.245 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.242+0000 7f901ee1e640 10 monclient(hunting): picked mon.noname-c con 0x7f90180b4d60 addr v1:172.21.15.167:6789/0 2024-03-25T18:28:39.245 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.242+0000 7f901ee1e640 10 monclient(hunting): picked mon.noname-a con 0x7f90180b1730 addr v1:172.21.15.155:6789/0 2024-03-25T18:28:39.245 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.242+0000 7f901ee1e640 10 monclient(hunting): picked mon.noname-b con 0x7f901814be90 addr v1:172.21.15.155:6790/0 2024-03-25T18:28:39.245 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.242+0000 7f901ee1e640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90180af940 con 0x7f90180b1730 2024-03-25T18:28:39.245 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.242+0000 7f901ee1e640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90180b0a70 con 0x7f90180b4d60 2024-03-25T18:28:39.245 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.242+0000 7f901ee1e640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90180b0d50 con 0x7f901814be90 2024-03-25T18:28:39.245 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.242+0000 7f901ee1e640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:39.245 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.242+0000 7f901ee1e640 10 monclient(hunting): authenticate will time out at 7765.552691s 2024-03-25T18:28:39.252 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.251+0000 7f9017fff640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f90180b1730 0x7f90180b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:46544/0 (socket says 172.21.15.155:46544) 2024-03-25T18:28:39.252 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.251+0000 7f9017fff640 1 -- 172.21.15.155:0/1419373732 learned_addr learned my addr 172.21.15.155:0/1419373732 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:28:39.277 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.276+0000 7f90177fe640 1 -- 172.21.15.155:0/1419373732 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4054808812 0 0) 0x7f90180af940 con 0x7f90180b1730 2024-03-25T18:28:39.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.276+0000 7f90177fe640 10 monclient(hunting): _negotiate 2024-03-25T18:28:39.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.276+0000 7f90177fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:39.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.276+0000 7f90177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:39.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.276+0000 7f90177fe640 10 monclient(hunting): my global_id is 4395 2024-03-25T18:28:39.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.276+0000 7f90177fe640 1 -- 172.21.15.155:0/1419373732 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9008003520 con 0x7f90180b1730 2024-03-25T18:28:39.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.278+0000 7f90177fe640 1 -- 172.21.15.155:0/1419373732 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 837296536 0 0) 0x7f90180b0a70 con 0x7f90180b4d60 2024-03-25T18:28:39.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.278+0000 7f90177fe640 10 monclient(hunting): _negotiate 2024-03-25T18:28:39.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.278+0000 7f90177fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:39.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.278+0000 7f90177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:39.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.278+0000 7f90177fe640 10 monclient(hunting): my global_id is 4390 2024-03-25T18:28:39.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.278+0000 7f90177fe640 1 -- 172.21.15.155:0/1419373732 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90180af940 con 0x7f90180b4d60 2024-03-25T18:28:39.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.287+0000 7f90177fe640 1 -- 172.21.15.155:0/1419373732 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3549091223 0 0) 0x7f90180af940 con 0x7f90180b4d60 2024-03-25T18:28:39.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.287+0000 7f90177fe640 1 -- 172.21.15.155:0/1419373732 >> v1:172.21.15.155:6790/0 conn(0x7f901814be90 legacy=0x7f901814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:39.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.287+0000 7f90177fe640 1 -- 172.21.15.155:0/1419373732 >> v1:172.21.15.155:6789/0 conn(0x7f90180b1730 legacy=0x7f90180b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:39.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.287+0000 7f90177fe640 10 monclient: _finish_hunting 0 2024-03-25T18:28:39.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.287+0000 7f90177fe640 1 monclient: found mon.noname-c 2024-03-25T18:28:39.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.287+0000 7f90177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:39.289 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.287+0000 7f90177fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.167:6789/0 2024-03-25T18:28:39.289 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.287+0000 7f90177fe640 1 -- 172.21.15.155:0/1419373732 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90180b1030 con 0x7f90180b4d60 2024-03-25T18:28:39.289 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.287+0000 7f90177fe640 10 monclient: _finish_auth 0 2024-03-25T18:28:39.289 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.287+0000 7f90177fe640 10 monclient: _check_auth_tickets 2024-03-25T18:28:39.289 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.287+0000 7f90177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:39.289 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.287+0000 7f90177fe640 1 -- 172.21.15.155:0/1419373732 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f900c001670 con 0x7f90180b4d60 2024-03-25T18:28:39.289 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.287+0000 7f901ee1e640 5 monclient: authenticate success, global_id 4390 2024-03-25T18:28:39.289 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.287+0000 7f901ee1e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:39.289 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.287+0000 7f90177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:39.289 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.287+0000 7f90177fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:28:39.289 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.287+0000 7f90177fe640 10 monclient: dump: 2024-03-25T18:28:39.289 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:28:39.289 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:28:39.289 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:39.290 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:39.290 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:39.290 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:28:39.290 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:28:39.290 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:28:39.290 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:28:39.290 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:28:39.290 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.287+0000 7f901ee1e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:39.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.296+0000 7f90177fe640 1 -- 172.21.15.155:0/1419373732 <== mon.1 v1:172.21.15.167:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f900c0017d0 con 0x7f90180b4d60 2024-03-25T18:28:39.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.296+0000 7f90177fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:39.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.296+0000 7f90177fe640 1 -- 172.21.15.155:0/1419373732 <== mon.1 v1:172.21.15.167:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f900c001ba0 con 0x7f90180b4d60 2024-03-25T18:28:39.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.296+0000 7f901ee1e640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:39.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.296+0000 7f901ee1e640 4 set_mon_vals no callback set 2024-03-25T18:28:39.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.296+0000 7f90177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:39.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.296+0000 7f90177fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:39.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.296+0000 7f90177fe640 10 monclient: dump: 2024-03-25T18:28:39.298 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:28:39.298 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:28:39.298 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:39.298 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:39.299 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:39.299 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:28:39.299 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:28:39.299 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:28:39.299 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:28:39.299 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:28:39.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.297+0000 7f901ee1e640 10 monclient: shutdown 2024-03-25T18:28:39.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.297+0000 7f901ee1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:39.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.297+0000 7f901ee1e640 1 -- 172.21.15.155:0/1419373732 >> v1:172.21.15.167:6789/0 conn(0x7f90180b4d60 legacy=0x7f901813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:39.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.297+0000 7f901ee1e640 1 -- 172.21.15.155:0/1419373732 shutdown_connections 2024-03-25T18:28:39.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.297+0000 7f901ee1e640 1 -- 172.21.15.155:0/1419373732 >> 172.21.15.155:0/1419373732 conn(0x7f90180617a0 msgr2=0x7f9018061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:39.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.297+0000 7f901ee1e640 1 -- 172.21.15.155:0/1419373732 shutdown_connections 2024-03-25T18:28:39.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.297+0000 7f901ee1e640 1 -- 172.21.15.155:0/1419373732 wait complete. 2024-03-25T18:28:39.302 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.298+0000 7f901ee1e640 10 monclient: build_initial_monmap 2024-03-25T18:28:39.302 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.298+0000 7f901ee1e640 10 monclient: monmap: 2024-03-25T18:28:39.302 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:28:39.302 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:39.302 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:28:39.302 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:28:39.302 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:39.302 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:28:39.302 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:28:39.302 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:28:39.302 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:28:39.302 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:28:39.302 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.298+0000 7f901ee1e640 1 librados: starting msgr at 2024-03-25T18:28:39.302 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.298+0000 7f901ee1e640 1 librados: starting objecter 2024-03-25T18:28:39.303 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.298+0000 7f901ee1e640 1 Processor -- start 2024-03-25T18:28:39.303 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.298+0000 7f901ee1e640 1 -- start start 2024-03-25T18:28:39.303 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.298+0000 7f901ee1e640 1 librados: setting wanted keys 2024-03-25T18:28:39.303 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.298+0000 7f901ee1e640 1 librados: calling monclient init 2024-03-25T18:28:39.303 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.298+0000 7f901ee1e640 10 monclient: init 2024-03-25T18:28:39.303 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.299+0000 7f901ee1e640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:39.303 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.299+0000 7f901ee1e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:28:39.303 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.299+0000 7f901ee1e640 10 monclient(hunting): picked mon.noname-c con 0x7f90180b1730 addr v1:172.21.15.155:6790/0 2024-03-25T18:28:39.303 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.299+0000 7f901ee1e640 10 monclient(hunting): picked mon.noname-b con 0x7f90180b4d60 addr v1:172.21.15.167:6789/0 2024-03-25T18:28:39.303 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.299+0000 7f901ee1e640 10 monclient(hunting): picked mon.noname-a con 0x7f901814be90 addr v1:172.21.15.155:6789/0 2024-03-25T18:28:39.303 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.299+0000 7f901ee1e640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90180aac60 con 0x7f901814be90 2024-03-25T18:28:39.303 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.299+0000 7f901ee1e640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90181dadd0 con 0x7f90180b4d60 2024-03-25T18:28:39.303 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.299+0000 7f901ee1e640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f901813bb50 con 0x7f90180b1730 2024-03-25T18:28:39.303 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.299+0000 7f901ee1e640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:39.304 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.299+0000 7f901ee1e640 10 monclient(hunting): authenticate will time out at 7765.609277s 2024-03-25T18:28:39.311 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.310+0000 7f9017fff640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f90180b4d60 0x7f90181db2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:38806/0 (socket says 172.21.15.155:38806) 2024-03-25T18:28:39.311 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.310+0000 7f9017fff640 1 -- 172.21.15.155:0/1083517679 learned_addr learned my addr 172.21.15.155:0/1083517679 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:28:39.312 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.312+0000 7f90157fa640 10 client.?.objecter ms_handle_connect 0x7f90180b4d60 2024-03-25T18:28:39.313 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.312+0000 7f90157fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:39.324 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.323+0000 7f90157fa640 10 client.?.objecter ms_handle_connect 0x7f90180b1730 2024-03-25T18:28:39.324 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.323+0000 7f90157fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:39.329 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.328+0000 7f90157fa640 1 -- 172.21.15.155:0/1083517679 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1798270174 0 0) 0x7f901813bb50 con 0x7f90180b1730 2024-03-25T18:28:39.329 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.328+0000 7f90157fa640 10 monclient(hunting): _negotiate 2024-03-25T18:28:39.329 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.328+0000 7f90157fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:39.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.328+0000 7f90157fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:39.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.328+0000 7f90157fa640 10 monclient(hunting): my global_id is 4403 2024-03-25T18:28:39.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.328+0000 7f90157fa640 1 -- 172.21.15.155:0/1083517679 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ffc004eb0 con 0x7f90180b1730 2024-03-25T18:28:39.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.329+0000 7f90157fa640 1 -- 172.21.15.155:0/1083517679 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4014377547 0 0) 0x7f90181dadd0 con 0x7f90180b4d60 2024-03-25T18:28:39.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.329+0000 7f90157fa640 10 monclient(hunting): _negotiate 2024-03-25T18:28:39.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.329+0000 7f90157fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:39.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.329+0000 7f90157fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:39.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.329+0000 7f90157fa640 10 monclient(hunting): my global_id is 4393 2024-03-25T18:28:39.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.329+0000 7f90157fa640 1 -- 172.21.15.155:0/1083517679 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f901813bb50 con 0x7f90180b4d60 2024-03-25T18:28:39.334 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.333+0000 7f90157fa640 10 client.?.objecter ms_handle_connect 0x7f901814be90 2024-03-25T18:28:39.334 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.333+0000 7f90157fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:39.340 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.339+0000 7f90157fa640 1 -- 172.21.15.155:0/1083517679 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2521089420 0 0) 0x7f901813bb50 con 0x7f90180b4d60 2024-03-25T18:28:39.340 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.339+0000 7f90157fa640 1 -- 172.21.15.155:0/1083517679 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f90181dadd0 con 0x7f90180b4d60 2024-03-25T18:28:39.340 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.339+0000 7f90157fa640 1 -- 172.21.15.155:0/1083517679 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f9004001950 con 0x7f90180b4d60 2024-03-25T18:28:39.340 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.339+0000 7f90157fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:39.340 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.339+0000 7f90157fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:28:39.340 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.339+0000 7f90157fa640 10 monclient(hunting): dump: 2024-03-25T18:28:39.340 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:28:39.341 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:28:39.341 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:39.341 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:39.341 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:39.341 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:28:39.341 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:28:39.341 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:28:39.341 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:28:39.341 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:28:39.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.339+0000 7f90157fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:39.346 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.345+0000 7f90157fa640 1 -- 172.21.15.155:0/1083517679 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2838652300 0 0) 0x7f8ffc004eb0 con 0x7f90180b1730 2024-03-25T18:28:39.346 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.345+0000 7f90157fa640 1 -- 172.21.15.155:0/1083517679 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f901813bb50 con 0x7f90180b1730 2024-03-25T18:28:39.346 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.345+0000 7f90157fa640 1 -- 172.21.15.155:0/1083517679 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f900c001ae0 con 0x7f90180b1730 2024-03-25T18:28:39.347 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.345+0000 7f90157fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:39.347 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.345+0000 7f90157fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:39.347 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.345+0000 7f90157fa640 10 monclient(hunting): dump: 2024-03-25T18:28:39.347 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:28:39.347 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:28:39.347 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:39.347 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:39.347 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:39.347 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:28:39.347 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:28:39.348 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:28:39.348 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:28:39.348 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:28:39.350 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.348+0000 7f90157fa640 1 -- 172.21.15.155:0/1083517679 <== mon.1 v1:172.21.15.167:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3363838471 0 0) 0x7f90181dadd0 con 0x7f90180b4d60 2024-03-25T18:28:39.350 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.348+0000 7f90157fa640 1 -- 172.21.15.155:0/1083517679 >> v1:172.21.15.155:6790/0 conn(0x7f90180b1730 legacy=0x7f901813b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:39.350 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.348+0000 7f90157fa640 1 -- 172.21.15.155:0/1083517679 >> v1:172.21.15.155:6789/0 conn(0x7f901814be90 legacy=0x7f9018137e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:39.350 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.348+0000 7f90157fa640 10 monclient: _finish_hunting 0 2024-03-25T18:28:39.351 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.348+0000 7f90157fa640 1 monclient: found mon.b 2024-03-25T18:28:39.351 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.348+0000 7f90157fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:39.351 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.348+0000 7f90157fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:28:39.351 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.348+0000 7f90157fa640 1 -- 172.21.15.155:0/1083517679 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9018138dd0 con 0x7f90180b4d60 2024-03-25T18:28:39.351 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.348+0000 7f90157fa640 10 monclient: _finish_auth 0 2024-03-25T18:28:39.351 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.348+0000 7f90157fa640 10 monclient: _check_auth_tickets 2024-03-25T18:28:39.351 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.348+0000 7f90157fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:39.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.348+0000 7f901ee1e640 5 monclient: authenticate success, global_id 4393 2024-03-25T18:28:39.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.348+0000 7f901ee1e640 10 monclient: _renew_subs 2024-03-25T18:28:39.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.348+0000 7f901ee1e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:28:39.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.348+0000 7f901ee1e640 1 -- 172.21.15.155:0/1083517679 --> v1:172.21.15.167:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90181dba00 con 0x7f90180b4d60 2024-03-25T18:28:39.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.348+0000 7f901ee1e640 10 client.4393.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:39.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.348+0000 7f901ee1e640 10 monclient: _renew_subs 2024-03-25T18:28:39.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.348+0000 7f901ee1e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:28:39.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.349+0000 7f901ee1e640 1 -- 172.21.15.155:0/1083517679 --> v1:172.21.15.167:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f90180a98c0 con 0x7f90180b4d60 2024-03-25T18:28:39.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.349+0000 7f901ee1e640 1 librados: init done 2024-03-25T18:28:39.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.350+0000 7f901ee1e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:39.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.350+0000 7f901ee1e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:39.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.350+0000 7f901ee1e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:28:39.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.350+0000 7f901ee1e640 1 -- 172.21.15.155:0/1083517679 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90181401f0 con 0x7f90180b4d60 2024-03-25T18:28:39.619 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.618+0000 7f90157fa640 1 -- 172.21.15.155:0/1083517679 <== mon.1 v1:172.21.15.167:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9004001d60 con 0x7f90180b4d60 2024-03-25T18:28:39.619 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.618+0000 7f90157fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:39.619 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.618+0000 7f90157fa640 1 -- 172.21.15.155:0/1083517679 <== mon.1 v1:172.21.15.167:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f90040049c0 con 0x7f90180b4d60 2024-03-25T18:28:39.619 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.618+0000 7f90157fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:39.619 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.618+0000 7f90157fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:39.619 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.618+0000 7f90157fa640 10 monclient: dump: 2024-03-25T18:28:39.619 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:28:39.620 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:28:39.620 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:39.620 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:39.620 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:39.620 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:28:39.620 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:28:39.620 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:28:39.620 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:28:39.620 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:28:39.620 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.618+0000 7f9016ffd640 4 set_mon_vals no callback set 2024-03-25T18:28:39.621 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.619+0000 7f90157fa640 1 -- 172.21.15.155:0/1083517679 <== mon.1 v1:172.21.15.167:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7f9004004be0 con 0x7f90180b4d60 2024-03-25T18:28:39.621 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.619+0000 7f901cbba640 10 monclient: get_auth_request con 0x7f8ffc06de90 auth_method 0 2024-03-25T18:28:39.621 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.620+0000 7f90157fa640 10 client.4393.objecter ms_handle_connect 0x7f8ffc06de90 2024-03-25T18:28:39.625 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.624+0000 7f90157fa640 1 -- 172.21.15.155:0/1083517679 <== mon.1 v1:172.21.15.167:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7f90040547e0 con 0x7f90180b4d60 2024-03-25T18:28:39.625 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.624+0000 7f90157fa640 10 client.4393.objecter ms_dispatch 0x7f9018000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:39.625 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.624+0000 7f90157fa640 3 client.4393.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:39.625 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.624+0000 7f90157fa640 3 client.4393.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:39.625 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.624+0000 7f90157fa640 20 client.4393.objecter dump_active .. 0 homeless 2024-03-25T18:28:39.626 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.625+0000 7f90157fa640 1 -- 172.21.15.155:0/1083517679 <== mon.1 v1:172.21.15.167:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9004050120 con 0x7f90180b4d60 2024-03-25T18:28:39.626 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.625+0000 7f90157fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:39.626 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.625+0000 7f90157fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:39.880 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.879+0000 7f901ee1e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:39.880 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.879+0000 7f901ee1e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:39.880 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.879+0000 7f901ee1e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:28:39.880 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.879+0000 7f901ee1e640 1 -- 172.21.15.155:0/1083517679 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f90180627b0 con 0x7f90180b4d60 2024-03-25T18:28:39.902 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.901+0000 7f90157fa640 1 -- 172.21.15.155:0/1083517679 <== mon.1 v1:172.21.15.167:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7f9004052220 con 0x7f90180b4d60 2024-03-25T18:28:39.902 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.901+0000 7f90157fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:39.902 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.901+0000 7f90157fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:39.903 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:28:39.903 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:39.905 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.904+0000 7f901ee1e640 10 librados: watch_flush enter 2024-03-25T18:28:39.905 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.904+0000 7f901ee1e640 10 librados: watch_flush exit 2024-03-25T18:28:39.905 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.904+0000 7f901ee1e640 20 client.4393.objecter shutdown clearing up homeless session... 2024-03-25T18:28:39.905 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.904+0000 7f901ee1e640 10 client.4393.objecter successfully canceled tick 2024-03-25T18:28:39.905 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.904+0000 7f901ee1e640 1 -- 172.21.15.155:0/1083517679 >> v1:172.21.15.167:6800/2919070503 conn(0x7f8ffc06de90 legacy=0x7f8ffc070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:39.906 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.904+0000 7f901ee1e640 10 monclient: shutdown 2024-03-25T18:28:39.906 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.904+0000 7f901ee1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:39.906 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.904+0000 7f901ee1e640 1 -- 172.21.15.155:0/1083517679 >> v1:172.21.15.167:6789/0 conn(0x7f90180b4d60 legacy=0x7f90181db2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:39.906 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.904+0000 7f901ee1e640 1 -- 172.21.15.155:0/1083517679 shutdown_connections 2024-03-25T18:28:39.906 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.904+0000 7f901ee1e640 1 -- 172.21.15.155:0/1083517679 >> 172.21.15.155:0/1083517679 conn(0x7f90180617a0 msgr2=0x7f901813c5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:39.906 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.904+0000 7f901ee1e640 1 -- 172.21.15.155:0/1083517679 shutdown_connections 2024-03-25T18:28:39.906 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.904+0000 7f901ee1e640 1 -- 172.21.15.155:0/1083517679 wait complete. 2024-03-25T18:28:39.906 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:39.904+0000 7f901ee1e640 1 librados: shutdown 2024-03-25T18:28:39.918 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:46.726 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:46.840 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.839+0000 7f5259857640 10 monclient: get_monmap_and_config 2024-03-25T18:28:46.841 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.839+0000 7f5259857640 10 monclient: build_initial_monmap 2024-03-25T18:28:46.841 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.839+0000 7f5259857640 10 monclient: monmap: 2024-03-25T18:28:46.841 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:28:46.841 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:28:46.841 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:28:46.840676+0000 2024-03-25T18:28:46.841 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:28:46.840676+0000 2024-03-25T18:28:46.841 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:46.841 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:28:46.841 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:28:46.841 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:28:46.841 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:28:46.842 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:28:46.842 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.840+0000 7f5259857640 1 Processor -- start 2024-03-25T18:28:46.842 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.840+0000 7f5259857640 1 -- start start 2024-03-25T18:28:46.843 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.840+0000 7f5259857640 10 monclient: init 2024-03-25T18:28:46.843 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.841+0000 7f5259857640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:46.843 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.841+0000 7f5259857640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:28:46.843 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.841+0000 7f5259857640 10 monclient(hunting): picked mon.noname-a con 0x7f52540b4d60 addr v1:172.21.15.155:6789/0 2024-03-25T18:28:46.843 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.841+0000 7f5259857640 10 monclient(hunting): picked mon.noname-c con 0x7f52540b1730 addr v1:172.21.15.167:6789/0 2024-03-25T18:28:46.843 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.841+0000 7f5259857640 10 monclient(hunting): picked mon.noname-b con 0x7f525414be90 addr v1:172.21.15.155:6790/0 2024-03-25T18:28:46.843 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.841+0000 7f5259857640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52540af940 con 0x7f52540b4d60 2024-03-25T18:28:46.843 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.841+0000 7f5259857640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52540b0a70 con 0x7f52540b1730 2024-03-25T18:28:46.843 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.841+0000 7f5259857640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52540b0d50 con 0x7f525414be90 2024-03-25T18:28:46.843 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.841+0000 7f5259857640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:46.843 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.841+0000 7f5259857640 10 monclient(hunting): authenticate will time out at 7773.151193s 2024-03-25T18:28:46.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.852+0000 7f524bfff640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f52540b1730 0x7f52540b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:53598/0 (socket says 172.21.15.155:53598) 2024-03-25T18:28:46.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.852+0000 7f524bfff640 1 -- 172.21.15.155:0/3943134081 learned_addr learned my addr 172.21.15.155:0/3943134081 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:28:46.876 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.875+0000 7f52527fc640 1 -- 172.21.15.155:0/3943134081 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1276939092 0 0) 0x7f52540b0a70 con 0x7f52540b1730 2024-03-25T18:28:46.877 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.875+0000 7f52527fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:46.877 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.875+0000 7f52527fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:46.877 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.875+0000 7f52527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:46.877 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.875+0000 7f52527fc640 10 monclient(hunting): my global_id is 4396 2024-03-25T18:28:46.877 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.875+0000 7f52527fc640 1 -- 172.21.15.155:0/3943134081 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5240003520 con 0x7f52540b1730 2024-03-25T18:28:46.887 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.886+0000 7f52527fc640 1 -- 172.21.15.155:0/3943134081 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1267037263 0 0) 0x7f52540b0d50 con 0x7f525414be90 2024-03-25T18:28:46.887 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.886+0000 7f52527fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:46.887 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.886+0000 7f52527fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:46.887 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.886+0000 7f52527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:46.888 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.886+0000 7f52527fc640 10 monclient(hunting): my global_id is 4406 2024-03-25T18:28:46.888 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.886+0000 7f52527fc640 1 -- 172.21.15.155:0/3943134081 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52540b0a70 con 0x7f525414be90 2024-03-25T18:28:46.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.893+0000 7f52527fc640 1 -- 172.21.15.155:0/3943134081 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1656510927 0 0) 0x7f52540af940 con 0x7f52540b4d60 2024-03-25T18:28:46.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.893+0000 7f52527fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:46.895 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.893+0000 7f52527fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:46.895 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.893+0000 7f52527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:46.895 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.893+0000 7f52527fc640 10 monclient(hunting): my global_id is 4401 2024-03-25T18:28:46.895 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.893+0000 7f52527fc640 1 -- 172.21.15.155:0/3943134081 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52540b0d50 con 0x7f52540b4d60 2024-03-25T18:28:46.900 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.899+0000 7f52527fc640 1 -- 172.21.15.155:0/3943134081 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3335335074 0 0) 0x7f5240003520 con 0x7f52540b1730 2024-03-25T18:28:46.901 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.899+0000 7f52527fc640 1 -- 172.21.15.155:0/3943134081 >> v1:172.21.15.155:6790/0 conn(0x7f525414be90 legacy=0x7f525414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:46.901 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.899+0000 7f52527fc640 1 -- 172.21.15.155:0/3943134081 >> v1:172.21.15.155:6789/0 conn(0x7f52540b4d60 legacy=0x7f525413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:46.901 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.899+0000 7f52527fc640 10 monclient: _finish_hunting 0 2024-03-25T18:28:46.901 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.899+0000 7f52527fc640 1 monclient: found mon.noname-c 2024-03-25T18:28:46.901 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.899+0000 7f52527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:46.901 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.899+0000 7f52527fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.167:6789/0 2024-03-25T18:28:46.901 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.899+0000 7f52527fc640 1 -- 172.21.15.155:0/3943134081 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52540b1030 con 0x7f52540b1730 2024-03-25T18:28:46.901 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.899+0000 7f52527fc640 10 monclient: _finish_auth 0 2024-03-25T18:28:46.901 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.899+0000 7f52527fc640 10 monclient: _check_auth_tickets 2024-03-25T18:28:46.901 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.899+0000 7f52527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:46.901 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.899+0000 7f52527fc640 1 -- 172.21.15.155:0/3943134081 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f523c001690 con 0x7f52540b1730 2024-03-25T18:28:46.901 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.899+0000 7f5259857640 5 monclient: authenticate success, global_id 4396 2024-03-25T18:28:46.901 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.899+0000 7f5259857640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:46.902 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.899+0000 7f52527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:46.902 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.899+0000 7f52527fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:28:46.902 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.899+0000 7f52527fc640 10 monclient: dump: 2024-03-25T18:28:46.902 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:28:46.902 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:28:46.902 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:46.902 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:46.902 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:46.902 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:28:46.902 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:28:46.902 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:28:46.902 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:28:46.902 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:28:46.902 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.899+0000 7f5259857640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:46.919 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.917+0000 7f52527fc640 1 -- 172.21.15.155:0/3943134081 <== mon.1 v1:172.21.15.167:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f523c0017f0 con 0x7f52540b1730 2024-03-25T18:28:46.919 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.917+0000 7f52527fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:46.919 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.917+0000 7f52527fc640 1 -- 172.21.15.155:0/3943134081 <== mon.1 v1:172.21.15.167:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f523c001c00 con 0x7f52540b1730 2024-03-25T18:28:46.919 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.917+0000 7f52527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:46.919 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.917+0000 7f52527fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:46.919 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.917+0000 7f52527fc640 10 monclient: dump: 2024-03-25T18:28:46.919 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:28:46.919 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:28:46.919 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:46.919 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:46.919 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:46.919 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:28:46.919 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:28:46.919 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:28:46.920 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:28:46.920 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:28:46.920 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.917+0000 7f5259857640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:46.920 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.917+0000 7f5259857640 4 set_mon_vals no callback set 2024-03-25T18:28:46.922 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.918+0000 7f5259857640 10 monclient: shutdown 2024-03-25T18:28:46.922 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.918+0000 7f5259857640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:46.922 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.918+0000 7f5259857640 1 -- 172.21.15.155:0/3943134081 >> v1:172.21.15.167:6789/0 conn(0x7f52540b1730 legacy=0x7f52540b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:46.922 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.918+0000 7f5259857640 1 -- 172.21.15.155:0/3943134081 shutdown_connections 2024-03-25T18:28:46.922 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.918+0000 7f5259857640 1 -- 172.21.15.155:0/3943134081 >> 172.21.15.155:0/3943134081 conn(0x7f52540617a0 msgr2=0x7f5254061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:46.922 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.918+0000 7f5259857640 1 -- 172.21.15.155:0/3943134081 shutdown_connections 2024-03-25T18:28:46.923 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.918+0000 7f5259857640 1 -- 172.21.15.155:0/3943134081 wait complete. 2024-03-25T18:28:46.923 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.919+0000 7f5259857640 10 monclient: build_initial_monmap 2024-03-25T18:28:46.923 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.919+0000 7f5259857640 10 monclient: monmap: 2024-03-25T18:28:46.923 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:28:46.923 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:46.923 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:28:46.923 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:28:46.923 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:46.923 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:28:46.923 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:28:46.923 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:28:46.923 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:28:46.923 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:28:46.923 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.919+0000 7f5259857640 1 librados: starting msgr at 2024-03-25T18:28:46.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.919+0000 7f5259857640 1 librados: starting objecter 2024-03-25T18:28:46.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.920+0000 7f5259857640 1 Processor -- start 2024-03-25T18:28:46.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.920+0000 7f5259857640 1 -- start start 2024-03-25T18:28:46.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.920+0000 7f5259857640 1 librados: setting wanted keys 2024-03-25T18:28:46.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.920+0000 7f5259857640 1 librados: calling monclient init 2024-03-25T18:28:46.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.920+0000 7f5259857640 10 monclient: init 2024-03-25T18:28:46.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.920+0000 7f5259857640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:46.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.920+0000 7f5259857640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:28:46.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.921+0000 7f5259857640 10 monclient(hunting): picked mon.noname-b con 0x7f52540b1730 addr v1:172.21.15.167:6789/0 2024-03-25T18:28:46.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.921+0000 7f5259857640 10 monclient(hunting): picked mon.noname-c con 0x7f52540b4d60 addr v1:172.21.15.155:6790/0 2024-03-25T18:28:46.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.921+0000 7f5259857640 10 monclient(hunting): picked mon.noname-a con 0x7f525414be90 addr v1:172.21.15.155:6789/0 2024-03-25T18:28:46.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.921+0000 7f5259857640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f525413e960 con 0x7f525414be90 2024-03-25T18:28:46.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.921+0000 7f5259857640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5254141b30 con 0x7f52540b1730 2024-03-25T18:28:46.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.921+0000 7f5259857640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5254137b80 con 0x7f52540b4d60 2024-03-25T18:28:46.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.921+0000 7f5259857640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:46.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.921+0000 7f5259857640 10 monclient(hunting): authenticate will time out at 7773.231225s 2024-03-25T18:28:46.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.927+0000 7f52537fe640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f525414be90 0x7f52541e2f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:56912/0 (socket says 172.21.15.155:56912) 2024-03-25T18:28:46.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.927+0000 7f52537fe640 1 -- 172.21.15.155:0/1193715090 learned_addr learned my addr 172.21.15.155:0/1193715090 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:28:46.933 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.932+0000 7f5250ff9640 10 client.?.objecter ms_handle_connect 0x7f525414be90 2024-03-25T18:28:46.933 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.932+0000 7f5250ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:46.939 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.938+0000 7f5250ff9640 10 client.?.objecter ms_handle_connect 0x7f52540b4d60 2024-03-25T18:28:46.939 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.938+0000 7f5250ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:46.940 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.939+0000 7f5250ff9640 10 client.?.objecter ms_handle_connect 0x7f52540b1730 2024-03-25T18:28:46.940 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.939+0000 7f5250ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:46.950 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.949+0000 7f5250ff9640 1 -- 172.21.15.155:0/1193715090 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2739526531 0 0) 0x7f525413e960 con 0x7f525414be90 2024-03-25T18:28:46.951 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.949+0000 7f5250ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:28:46.951 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.949+0000 7f5250ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:46.951 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.949+0000 7f5250ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:46.951 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.949+0000 7f5250ff9640 10 monclient(hunting): my global_id is 4404 2024-03-25T18:28:46.951 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.949+0000 7f5250ff9640 1 -- 172.21.15.155:0/1193715090 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5234004eb0 con 0x7f525414be90 2024-03-25T18:28:46.956 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.955+0000 7f5250ff9640 1 -- 172.21.15.155:0/1193715090 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 643060175 0 0) 0x7f5254137b80 con 0x7f52540b4d60 2024-03-25T18:28:46.956 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.955+0000 7f5250ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:28:46.956 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.955+0000 7f5250ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:46.956 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.955+0000 7f5250ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:46.956 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.955+0000 7f5250ff9640 10 monclient(hunting): my global_id is 4409 2024-03-25T18:28:46.956 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.955+0000 7f5250ff9640 1 -- 172.21.15.155:0/1193715090 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f525413e960 con 0x7f52540b4d60 2024-03-25T18:28:46.964 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.963+0000 7f5250ff9640 1 -- 172.21.15.155:0/1193715090 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3531899864 0 0) 0x7f5234004eb0 con 0x7f525414be90 2024-03-25T18:28:46.964 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.963+0000 7f5250ff9640 1 -- 172.21.15.155:0/1193715090 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5254137b80 con 0x7f525414be90 2024-03-25T18:28:46.964 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.963+0000 7f5250ff9640 1 -- 172.21.15.155:0/1193715090 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f524c001c90 con 0x7f525414be90 2024-03-25T18:28:46.964 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.963+0000 7f5250ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:46.964 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.963+0000 7f5250ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:28:46.964 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.963+0000 7f5250ff9640 10 monclient(hunting): dump: 2024-03-25T18:28:46.965 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:28:46.965 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:28:46.965 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:46.965 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:46.965 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:46.965 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:28:46.965 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:28:46.965 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:28:46.965 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:28:46.965 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:28:46.965 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.963+0000 7f5250ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:46.966 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.964+0000 7f5250ff9640 1 -- 172.21.15.155:0/1193715090 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3006310242 0 0) 0x7f5254141b30 con 0x7f52540b1730 2024-03-25T18:28:46.966 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.964+0000 7f5250ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:28:46.966 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.964+0000 7f5250ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:46.966 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.964+0000 7f5250ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:46.966 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.964+0000 7f5250ff9640 10 monclient(hunting): my global_id is 4399 2024-03-25T18:28:46.966 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.964+0000 7f5250ff9640 1 -- 172.21.15.155:0/1193715090 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5234004eb0 con 0x7f52540b1730 2024-03-25T18:28:46.970 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.969+0000 7f5250ff9640 1 -- 172.21.15.155:0/1193715090 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 53979630 0 0) 0x7f525413e960 con 0x7f52540b4d60 2024-03-25T18:28:46.970 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.969+0000 7f5250ff9640 1 -- 172.21.15.155:0/1193715090 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5254141b30 con 0x7f52540b4d60 2024-03-25T18:28:46.971 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.969+0000 7f5250ff9640 1 -- 172.21.15.155:0/1193715090 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f523c001b00 con 0x7f52540b4d60 2024-03-25T18:28:46.971 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.969+0000 7f5250ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:46.971 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.969+0000 7f5250ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:46.971 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.969+0000 7f5250ff9640 10 monclient(hunting): dump: 2024-03-25T18:28:46.971 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:28:46.971 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:28:46.971 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:46.971 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:46.971 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:46.971 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:28:46.971 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:28:46.971 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:28:46.971 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:28:46.971 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:28:46.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.980+0000 7f5250ff9640 1 -- 172.21.15.155:0/1193715090 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1844091382 0 0) 0x7f5254137b80 con 0x7f525414be90 2024-03-25T18:28:46.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.980+0000 7f5250ff9640 1 -- 172.21.15.155:0/1193715090 >> v1:172.21.15.155:6790/0 conn(0x7f52540b4d60 legacy=0x7f52541dab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:46.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.980+0000 7f5250ff9640 1 -- 172.21.15.155:0/1193715090 >> v1:172.21.15.167:6789/0 conn(0x7f52540b1730 legacy=0x7f5254134db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:46.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.980+0000 7f5250ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:28:46.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.980+0000 7f5250ff9640 1 monclient: found mon.a 2024-03-25T18:28:46.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.980+0000 7f5250ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:46.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.980+0000 7f5250ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:28:46.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.980+0000 7f5250ff9640 1 -- 172.21.15.155:0/1193715090 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5254137d10 con 0x7f525414be90 2024-03-25T18:28:46.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.980+0000 7f5250ff9640 10 monclient: _finish_auth 0 2024-03-25T18:28:46.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.980+0000 7f5250ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:28:46.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.980+0000 7f5250ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:46.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.980+0000 7f5259857640 5 monclient: authenticate success, global_id 4404 2024-03-25T18:28:46.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.980+0000 7f5259857640 10 monclient: _renew_subs 2024-03-25T18:28:46.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.980+0000 7f5259857640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:28:46.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.980+0000 7f5259857640 1 -- 172.21.15.155:0/1193715090 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52541386c0 con 0x7f525414be90 2024-03-25T18:28:46.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.981+0000 7f5259857640 10 client.4404.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:46.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.981+0000 7f5259857640 10 monclient: _renew_subs 2024-03-25T18:28:46.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.981+0000 7f5259857640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:28:46.984 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.981+0000 7f5259857640 1 -- 172.21.15.155:0/1193715090 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f52541356a0 con 0x7f525414be90 2024-03-25T18:28:46.984 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.981+0000 7f5259857640 1 librados: init done 2024-03-25T18:28:46.984 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.981+0000 7f5259857640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:46.984 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.982+0000 7f5259857640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:46.984 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.982+0000 7f5259857640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:28:46.984 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:46.982+0000 7f5259857640 1 -- 172.21.15.155:0/1193715090 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f525413ef20 con 0x7f525414be90 2024-03-25T18:28:47.207 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:47.206+0000 7f5250ff9640 1 -- 172.21.15.155:0/1193715090 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f524c003970 con 0x7f525414be90 2024-03-25T18:28:47.207 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:47.206+0000 7f5250ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:47.207 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:47.206+0000 7f5251ffb640 4 set_mon_vals no callback set 2024-03-25T18:28:47.300 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:47.299+0000 7f5250ff9640 1 -- 172.21.15.155:0/1193715090 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f524c003d80 con 0x7f525414be90 2024-03-25T18:28:47.300 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:47.299+0000 7f5250ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:47.300 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:47.299+0000 7f5250ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:47.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:47.299+0000 7f5250ff9640 10 monclient: dump: 2024-03-25T18:28:47.301 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:28:47.301 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:28:47.301 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:47.301 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:47.301 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:47.301 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:28:47.301 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:28:47.301 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:28:47.301 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:28:47.301 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:28:47.302 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:47.300+0000 7f5250ff9640 1 -- 172.21.15.155:0/1193715090 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7f524c003f60 con 0x7f525414be90 2024-03-25T18:28:47.302 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:47.300+0000 7f5252ffd640 10 monclient: get_auth_request con 0x7f523406dfb0 auth_method 0 2024-03-25T18:28:47.302 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:47.301+0000 7f5250ff9640 10 client.4404.objecter ms_handle_connect 0x7f523406dfb0 2024-03-25T18:28:47.306 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:47.305+0000 7f5250ff9640 1 -- 172.21.15.155:0/1193715090 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7f524c054a50 con 0x7f525414be90 2024-03-25T18:28:47.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:47.305+0000 7f5250ff9640 10 client.4404.objecter ms_dispatch 0x7f5254000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:47.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:47.305+0000 7f5250ff9640 3 client.4404.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:47.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:47.305+0000 7f5250ff9640 3 client.4404.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:47.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:47.305+0000 7f5250ff9640 20 client.4404.objecter dump_active .. 0 homeless 2024-03-25T18:28:47.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:47.305+0000 7f5250ff9640 1 -- 172.21.15.155:0/1193715090 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f524c050560 con 0x7f525414be90 2024-03-25T18:28:47.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:47.305+0000 7f5250ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:47.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:47.305+0000 7f5250ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:47.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:47.544+0000 7f5259857640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:47.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:47.544+0000 7f5259857640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:47.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:47.544+0000 7f5259857640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:28:47.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:47.544+0000 7f5259857640 1 -- 172.21.15.155:0/1193715090 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f525413f0d0 con 0x7f525414be90 2024-03-25T18:28:47.595 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:47.593+0000 7f5250ff9640 1 -- 172.21.15.155:0/1193715090 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7f524c052490 con 0x7f525414be90 2024-03-25T18:28:47.595 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:47.593+0000 7f5250ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:47.595 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:47.593+0000 7f5250ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:47.596 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:28:47.596 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:47.597 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:47.595+0000 7f5259857640 10 librados: watch_flush enter 2024-03-25T18:28:47.597 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:47.596+0000 7f5259857640 10 librados: watch_flush exit 2024-03-25T18:28:47.597 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:47.596+0000 7f5259857640 20 client.4404.objecter shutdown clearing up homeless session... 2024-03-25T18:28:47.598 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:47.596+0000 7f5259857640 10 client.4404.objecter successfully canceled tick 2024-03-25T18:28:47.598 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:47.596+0000 7f5259857640 1 -- 172.21.15.155:0/1193715090 >> v1:172.21.15.167:6800/2919070503 conn(0x7f523406dfb0 legacy=0x7f5234070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:47.598 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:47.596+0000 7f5259857640 10 monclient: shutdown 2024-03-25T18:28:47.598 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:47.596+0000 7f5259857640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:47.598 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:47.596+0000 7f5259857640 1 -- 172.21.15.155:0/1193715090 >> v1:172.21.15.155:6789/0 conn(0x7f525414be90 legacy=0x7f52541e2f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:47.598 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:47.596+0000 7f5259857640 1 -- 172.21.15.155:0/1193715090 shutdown_connections 2024-03-25T18:28:47.598 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:47.596+0000 7f5259857640 1 -- 172.21.15.155:0/1193715090 >> 172.21.15.155:0/1193715090 conn(0x7f52540617a0 msgr2=0x7f52540b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:47.598 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:47.596+0000 7f5259857640 1 -- 172.21.15.155:0/1193715090 shutdown_connections 2024-03-25T18:28:47.598 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:47.596+0000 7f5259857640 1 -- 172.21.15.155:0/1193715090 wait complete. 2024-03-25T18:28:47.598 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:47.596+0000 7f5259857640 1 librados: shutdown 2024-03-25T18:28:47.609 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:54.416 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:54.533 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.531+0000 7f23bfc25640 10 monclient: get_monmap_and_config 2024-03-25T18:28:54.533 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.531+0000 7f23bfc25640 10 monclient: build_initial_monmap 2024-03-25T18:28:54.533 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.532+0000 7f23bfc25640 10 monclient: monmap: 2024-03-25T18:28:54.533 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:28:54.533 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:28:54.533 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:28:54.532940+0000 2024-03-25T18:28:54.533 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:28:54.532940+0000 2024-03-25T18:28:54.533 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:54.533 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:28:54.533 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:28:54.534 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:28:54.534 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:28:54.534 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:28:54.535 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.532+0000 7f23bfc25640 1 Processor -- start 2024-03-25T18:28:54.535 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.533+0000 7f23bfc25640 1 -- start start 2024-03-25T18:28:54.535 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.533+0000 7f23bfc25640 10 monclient: init 2024-03-25T18:28:54.535 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.534+0000 7f23bfc25640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:54.535 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.534+0000 7f23bfc25640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:28:54.535 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.534+0000 7f23bfc25640 10 monclient(hunting): picked mon.noname-b con 0x7f23b80a98f0 addr v1:172.21.15.155:6790/0 2024-03-25T18:28:54.535 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.534+0000 7f23bfc25640 10 monclient(hunting): picked mon.noname-c con 0x7f23b813f710 addr v1:172.21.15.167:6789/0 2024-03-25T18:28:54.536 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.534+0000 7f23bfc25640 10 monclient(hunting): picked mon.noname-a con 0x7f23b81430c0 addr v1:172.21.15.155:6789/0 2024-03-25T18:28:54.536 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.534+0000 7f23bfc25640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23b8142520 con 0x7f23b81430c0 2024-03-25T18:28:54.536 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.534+0000 7f23bfc25640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23b80b28a0 con 0x7f23b813f710 2024-03-25T18:28:54.536 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.534+0000 7f23bfc25640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23b80b2a30 con 0x7f23b80a98f0 2024-03-25T18:28:54.536 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.534+0000 7f23bfc25640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:54.536 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.534+0000 7f23bfc25640 10 monclient(hunting): authenticate will time out at 7780.844032s 2024-03-25T18:28:54.556 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.555+0000 7f23be1c2640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f23b81430c0 0x7f23b814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:33458/0 (socket says 172.21.15.155:33458) 2024-03-25T18:28:54.557 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.555+0000 7f23be1c2640 1 -- 172.21.15.155:0/3645662513 learned_addr learned my addr 172.21.15.155:0/3645662513 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:28:54.567 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.566+0000 7f23bc9bf640 1 -- 172.21.15.155:0/3645662513 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3850485288 0 0) 0x7f23b8142520 con 0x7f23b81430c0 2024-03-25T18:28:54.567 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.566+0000 7f23bc9bf640 10 monclient(hunting): _negotiate 2024-03-25T18:28:54.567 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.566+0000 7f23bc9bf640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:54.567 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.566+0000 7f23bc9bf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:54.567 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.566+0000 7f23bc9bf640 10 monclient(hunting): my global_id is 4407 2024-03-25T18:28:54.567 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.566+0000 7f23bc9bf640 1 -- 172.21.15.155:0/3645662513 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23a4003520 con 0x7f23b81430c0 2024-03-25T18:28:54.574 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.573+0000 7f23bc9bf640 1 -- 172.21.15.155:0/3645662513 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4155127247 0 0) 0x7f23b80b28a0 con 0x7f23b813f710 2024-03-25T18:28:54.574 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.573+0000 7f23bc9bf640 10 monclient(hunting): _negotiate 2024-03-25T18:28:54.574 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.573+0000 7f23bc9bf640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:54.574 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.573+0000 7f23bc9bf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:54.574 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.573+0000 7f23bc9bf640 10 monclient(hunting): my global_id is 4402 2024-03-25T18:28:54.575 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.573+0000 7f23bc9bf640 1 -- 172.21.15.155:0/3645662513 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23b8142520 con 0x7f23b813f710 2024-03-25T18:28:54.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.577+0000 7f23bc9bf640 1 -- 172.21.15.155:0/3645662513 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2078659396 0 0) 0x7f23b80b2a30 con 0x7f23b80a98f0 2024-03-25T18:28:54.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.577+0000 7f23bc9bf640 10 monclient(hunting): _negotiate 2024-03-25T18:28:54.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.577+0000 7f23bc9bf640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:54.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.577+0000 7f23bc9bf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:54.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.577+0000 7f23bc9bf640 10 monclient(hunting): my global_id is 4412 2024-03-25T18:28:54.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.577+0000 7f23bc9bf640 1 -- 172.21.15.155:0/3645662513 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23b80b28a0 con 0x7f23b80a98f0 2024-03-25T18:28:54.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.581+0000 7f23bc9bf640 1 -- 172.21.15.155:0/3645662513 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3566308277 0 0) 0x7f23a4003520 con 0x7f23b81430c0 2024-03-25T18:28:54.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.581+0000 7f23bc9bf640 1 -- 172.21.15.155:0/3645662513 >> v1:172.21.15.155:6790/0 conn(0x7f23b80a98f0 legacy=0x7f23b80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:54.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.581+0000 7f23bc9bf640 1 -- 172.21.15.155:0/3645662513 >> v1:172.21.15.167:6789/0 conn(0x7f23b813f710 legacy=0x7f23b81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:54.583 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.581+0000 7f23bc9bf640 10 monclient: _finish_hunting 0 2024-03-25T18:28:54.583 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.581+0000 7f23bc9bf640 1 monclient: found mon.noname-a 2024-03-25T18:28:54.583 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.581+0000 7f23bc9bf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:54.583 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.581+0000 7f23bc9bf640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-25T18:28:54.583 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.581+0000 7f23bc9bf640 1 -- 172.21.15.155:0/3645662513 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23b80b44c0 con 0x7f23b81430c0 2024-03-25T18:28:54.583 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.581+0000 7f23bc9bf640 10 monclient: _finish_auth 0 2024-03-25T18:28:54.583 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.581+0000 7f23bc9bf640 10 monclient: _check_auth_tickets 2024-03-25T18:28:54.583 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.581+0000 7f23bc9bf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:54.584 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.581+0000 7f23bc9bf640 1 -- 172.21.15.155:0/3645662513 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f23b0001a10 con 0x7f23b81430c0 2024-03-25T18:28:54.584 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.581+0000 7f23bc9bf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:54.584 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.581+0000 7f23bc9bf640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:28:54.584 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.581+0000 7f23bc9bf640 10 monclient: dump: 2024-03-25T18:28:54.584 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:28:54.584 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:28:54.584 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:54.584 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:54.584 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:54.584 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:28:54.584 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:28:54.584 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:28:54.584 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:28:54.585 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:28:54.585 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.581+0000 7f23bfc25640 5 monclient: authenticate success, global_id 4407 2024-03-25T18:28:54.585 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.581+0000 7f23bfc25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:54.588 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.587+0000 7f23bc9bf640 1 -- 172.21.15.155:0/3645662513 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23b0001b70 con 0x7f23b81430c0 2024-03-25T18:28:54.588 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.587+0000 7f23bc9bf640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:54.588 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.587+0000 7f23bc9bf640 1 -- 172.21.15.155:0/3645662513 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f23b0001f80 con 0x7f23b81430c0 2024-03-25T18:28:54.589 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.587+0000 7f23bc9bf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:54.589 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.587+0000 7f23bc9bf640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:54.589 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.587+0000 7f23bc9bf640 10 monclient: dump: 2024-03-25T18:28:54.589 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:28:54.589 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:28:54.589 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:54.589 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:54.589 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:54.589 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:28:54.589 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:28:54.589 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:28:54.589 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:28:54.589 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:28:54.590 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.587+0000 7f23bfc25640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:54.590 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.587+0000 7f23bfc25640 4 set_mon_vals no callback set 2024-03-25T18:28:54.592 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.588+0000 7f23bfc25640 10 monclient: shutdown 2024-03-25T18:28:54.592 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.588+0000 7f23bfc25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:54.592 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.588+0000 7f23bfc25640 1 -- 172.21.15.155:0/3645662513 >> v1:172.21.15.155:6789/0 conn(0x7f23b81430c0 legacy=0x7f23b814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:54.592 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.588+0000 7f23bfc25640 1 -- 172.21.15.155:0/3645662513 shutdown_connections 2024-03-25T18:28:54.592 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.588+0000 7f23bfc25640 1 -- 172.21.15.155:0/3645662513 >> 172.21.15.155:0/3645662513 conn(0x7f23b80b7ea0 msgr2=0x7f23b80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:54.592 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.588+0000 7f23bfc25640 1 -- 172.21.15.155:0/3645662513 shutdown_connections 2024-03-25T18:28:54.592 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.588+0000 7f23bfc25640 1 -- 172.21.15.155:0/3645662513 wait complete. 2024-03-25T18:28:54.592 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.589+0000 7f23bfc25640 10 monclient: build_initial_monmap 2024-03-25T18:28:54.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.589+0000 7f23bfc25640 10 monclient: monmap: 2024-03-25T18:28:54.593 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:28:54.593 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:54.593 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:28:54.593 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:28:54.593 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:54.593 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:28:54.593 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:28:54.593 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:28:54.593 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:28:54.593 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:28:54.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.590+0000 7f23bfc25640 1 librados: starting msgr at 2024-03-25T18:28:54.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.590+0000 7f23bfc25640 1 librados: starting objecter 2024-03-25T18:28:54.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.590+0000 7f23bfc25640 1 Processor -- start 2024-03-25T18:28:54.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.590+0000 7f23bfc25640 1 -- start start 2024-03-25T18:28:54.594 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.590+0000 7f23bfc25640 1 librados: setting wanted keys 2024-03-25T18:28:54.594 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.590+0000 7f23bfc25640 1 librados: calling monclient init 2024-03-25T18:28:54.594 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.590+0000 7f23bfc25640 10 monclient: init 2024-03-25T18:28:54.594 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.590+0000 7f23bfc25640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:54.594 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.590+0000 7f23bfc25640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:28:54.594 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.590+0000 7f23bfc25640 10 monclient(hunting): picked mon.noname-c con 0x7f23b80a98f0 addr v1:172.21.15.155:6790/0 2024-03-25T18:28:54.594 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.591+0000 7f23bfc25640 10 monclient(hunting): picked mon.noname-b con 0x7f23b813f710 addr v1:172.21.15.167:6789/0 2024-03-25T18:28:54.594 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.591+0000 7f23bfc25640 10 monclient(hunting): picked mon.noname-a con 0x7f23b81430c0 addr v1:172.21.15.155:6789/0 2024-03-25T18:28:54.594 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.591+0000 7f23bfc25640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23b8136e20 con 0x7f23b81430c0 2024-03-25T18:28:54.594 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.591+0000 7f23bfc25640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23b80b7940 con 0x7f23b813f710 2024-03-25T18:28:54.594 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.591+0000 7f23bfc25640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23b81351a0 con 0x7f23b80a98f0 2024-03-25T18:28:54.594 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.591+0000 7f23bfc25640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:54.594 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.591+0000 7f23bfc25640 10 monclient(hunting): authenticate will time out at 7780.901040s 2024-03-25T18:28:54.614 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.613+0000 7f23bd9c1640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f23b80a98f0 0x7f23b81daa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:45410/0 (socket says 172.21.15.155:45410) 2024-03-25T18:28:54.614 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.613+0000 7f23bd9c1640 1 -- 172.21.15.155:0/940728773 learned_addr learned my addr 172.21.15.155:0/940728773 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:28:54.615 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.614+0000 7f23ae7fc640 10 client.?.objecter ms_handle_connect 0x7f23b80a98f0 2024-03-25T18:28:54.615 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.614+0000 7f23ae7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:54.618 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.617+0000 7f23ae7fc640 10 client.?.objecter ms_handle_connect 0x7f23b813f710 2024-03-25T18:28:54.618 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.617+0000 7f23ae7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:54.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.621+0000 7f23ae7fc640 1 -- 172.21.15.155:0/940728773 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1074116368 0 0) 0x7f23b80b7940 con 0x7f23b813f710 2024-03-25T18:28:54.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.621+0000 7f23ae7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:54.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.621+0000 7f23ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:54.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.621+0000 7f23ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:54.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.621+0000 7f23ae7fc640 10 monclient(hunting): my global_id is 4405 2024-03-25T18:28:54.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.621+0000 7f23ae7fc640 1 -- 172.21.15.155:0/940728773 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f239c004eb0 con 0x7f23b813f710 2024-03-25T18:28:54.625 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.624+0000 7f23ae7fc640 10 client.?.objecter ms_handle_connect 0x7f23b81430c0 2024-03-25T18:28:54.625 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.624+0000 7f23ae7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:54.625 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.624+0000 7f23ae7fc640 1 -- 172.21.15.155:0/940728773 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1773108070 0 0) 0x7f23b81351a0 con 0x7f23b80a98f0 2024-03-25T18:28:54.626 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.624+0000 7f23ae7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:54.626 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.624+0000 7f23ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:54.626 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.624+0000 7f23ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:54.626 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.624+0000 7f23ae7fc640 10 monclient(hunting): my global_id is 4415 2024-03-25T18:28:54.626 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.624+0000 7f23ae7fc640 1 -- 172.21.15.155:0/940728773 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23b80b7940 con 0x7f23b80a98f0 2024-03-25T18:28:54.635 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.634+0000 7f23ae7fc640 1 -- 172.21.15.155:0/940728773 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1402069699 0 0) 0x7f23b8136e20 con 0x7f23b81430c0 2024-03-25T18:28:54.635 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.634+0000 7f23ae7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:54.635 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.634+0000 7f23ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:54.635 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.634+0000 7f23ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:54.635 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.634+0000 7f23ae7fc640 10 monclient(hunting): my global_id is 4410 2024-03-25T18:28:54.635 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.634+0000 7f23ae7fc640 1 -- 172.21.15.155:0/940728773 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23b81351a0 con 0x7f23b81430c0 2024-03-25T18:28:54.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.636+0000 7f23ae7fc640 1 -- 172.21.15.155:0/940728773 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2968949258 0 0) 0x7f23b80b7940 con 0x7f23b80a98f0 2024-03-25T18:28:54.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.636+0000 7f23ae7fc640 1 -- 172.21.15.155:0/940728773 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23b8136e20 con 0x7f23b80a98f0 2024-03-25T18:28:54.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.636+0000 7f23ae7fc640 1 -- 172.21.15.155:0/940728773 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f23b40019f0 con 0x7f23b80a98f0 2024-03-25T18:28:54.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.636+0000 7f23ae7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:54.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.636+0000 7f23ae7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:28:54.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.636+0000 7f23ae7fc640 10 monclient(hunting): dump: 2024-03-25T18:28:54.638 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:28:54.638 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:28:54.638 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:54.638 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:54.638 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:54.638 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:28:54.638 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:28:54.638 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:28:54.638 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:28:54.638 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:28:54.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.636+0000 7f23ae7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:54.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.645+0000 7f23ae7fc640 1 -- 172.21.15.155:0/940728773 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1608708618 0 0) 0x7f239c004eb0 con 0x7f23b813f710 2024-03-25T18:28:54.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.645+0000 7f23ae7fc640 1 -- 172.21.15.155:0/940728773 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23b80b7940 con 0x7f23b813f710 2024-03-25T18:28:54.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.645+0000 7f23ae7fc640 1 -- 172.21.15.155:0/940728773 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f23a8001970 con 0x7f23b813f710 2024-03-25T18:28:54.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.645+0000 7f23ae7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:54.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.645+0000 7f23ae7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:54.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.645+0000 7f23ae7fc640 10 monclient(hunting): dump: 2024-03-25T18:28:54.648 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:28:54.648 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:28:54.648 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:54.648 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:54.649 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:54.649 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:28:54.649 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:28:54.649 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:28:54.649 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:28:54.649 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:28:54.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.645+0000 7f23ae7fc640 1 -- 172.21.15.155:0/940728773 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2573266439 0 0) 0x7f23b8136e20 con 0x7f23b80a98f0 2024-03-25T18:28:54.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.645+0000 7f23ae7fc640 1 -- 172.21.15.155:0/940728773 >> v1:172.21.15.167:6789/0 conn(0x7f23b813f710 legacy=0x7f23b81db1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:54.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.645+0000 7f23ae7fc640 1 -- 172.21.15.155:0/940728773 >> v1:172.21.15.155:6789/0 conn(0x7f23b81430c0 legacy=0x7f23b81e1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:54.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.645+0000 7f23ae7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:28:54.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.645+0000 7f23ae7fc640 1 monclient: found mon.c 2024-03-25T18:28:54.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.645+0000 7f23ae7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:54.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.645+0000 7f23ae7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:28:54.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.645+0000 7f23ae7fc640 1 -- 172.21.15.155:0/940728773 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23b8135330 con 0x7f23b80a98f0 2024-03-25T18:28:54.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.645+0000 7f23ae7fc640 10 monclient: _finish_auth 0 2024-03-25T18:28:54.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.645+0000 7f23ae7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:28:54.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.645+0000 7f23ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:54.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.645+0000 7f23bfc25640 5 monclient: authenticate success, global_id 4415 2024-03-25T18:28:54.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.645+0000 7f23bfc25640 10 monclient: _renew_subs 2024-03-25T18:28:54.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.645+0000 7f23bfc25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:28:54.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.645+0000 7f23bfc25640 1 -- 172.21.15.155:0/940728773 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23b8135560 con 0x7f23b80a98f0 2024-03-25T18:28:54.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.646+0000 7f23bfc25640 10 client.4415.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:54.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.646+0000 7f23bfc25640 10 monclient: _renew_subs 2024-03-25T18:28:54.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.646+0000 7f23bfc25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:28:54.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.646+0000 7f23bfc25640 1 -- 172.21.15.155:0/940728773 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f23b813b490 con 0x7f23b80a98f0 2024-03-25T18:28:54.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.646+0000 7f23bfc25640 1 librados: init done 2024-03-25T18:28:54.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.647+0000 7f23bfc25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:54.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.647+0000 7f23bfc25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:54.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.647+0000 7f23bfc25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:28:54.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.647+0000 7f23bfc25640 1 -- 172.21.15.155:0/940728773 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23b80b7730 con 0x7f23b80a98f0 2024-03-25T18:28:54.953 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.952+0000 7f23ae7fc640 1 -- 172.21.15.155:0/940728773 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23b4003610 con 0x7f23b80a98f0 2024-03-25T18:28:54.953 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.952+0000 7f23ae7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:54.953 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.952+0000 7f23ae7fc640 1 -- 172.21.15.155:0/940728773 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f23b4003920 con 0x7f23b80a98f0 2024-03-25T18:28:54.953 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.952+0000 7f23ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:54.953 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.952+0000 7f23ae7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:54.954 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.952+0000 7f23ae7fc640 10 monclient: dump: 2024-03-25T18:28:54.954 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:28:54.954 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:28:54.954 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:54.954 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:28:54.954 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:54.954 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:28:54.954 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:28:54.954 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:28:54.954 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:28:54.954 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:28:54.954 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.952+0000 7f23affff640 4 set_mon_vals no callback set 2024-03-25T18:28:54.955 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.953+0000 7f23ae7fc640 1 -- 172.21.15.155:0/940728773 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7f23b4003b40 con 0x7f23b80a98f0 2024-03-25T18:28:54.955 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.953+0000 7f23bd1c0640 10 monclient: get_auth_request con 0x7f239c06df70 auth_method 0 2024-03-25T18:28:54.955 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.954+0000 7f23ae7fc640 10 client.4415.objecter ms_handle_connect 0x7f239c06df70 2024-03-25T18:28:54.961 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.960+0000 7f23ae7fc640 1 -- 172.21.15.155:0/940728773 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7f23b4054770 con 0x7f23b80a98f0 2024-03-25T18:28:54.961 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.960+0000 7f23ae7fc640 10 client.4415.objecter ms_dispatch 0x7f23b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:54.961 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.960+0000 7f23ae7fc640 3 client.4415.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:54.961 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.960+0000 7f23ae7fc640 3 client.4415.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:54.961 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.960+0000 7f23ae7fc640 20 client.4415.objecter dump_active .. 0 homeless 2024-03-25T18:28:54.961 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.960+0000 7f23ae7fc640 1 -- 172.21.15.155:0/940728773 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f23b4050280 con 0x7f23b80a98f0 2024-03-25T18:28:54.961 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.960+0000 7f23ae7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:54.961 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:54.960+0000 7f23ae7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:55.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:55.195+0000 7f23bfc25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:55.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:55.195+0000 7f23bfc25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:55.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:55.195+0000 7f23bfc25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:28:55.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:55.195+0000 7f23bfc25640 1 -- 172.21.15.155:0/940728773 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23b80627b0 con 0x7f23b80a98f0 2024-03-25T18:28:55.303 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:55.302+0000 7f23ae7fc640 1 -- 172.21.15.155:0/940728773 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7f23b40521b0 con 0x7f23b80a98f0 2024-03-25T18:28:55.303 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:55.302+0000 7f23ae7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:55.303 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:55.302+0000 7f23ae7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:55.304 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:28:55.304 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:55.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:55.305+0000 7f23bfc25640 10 librados: watch_flush enter 2024-03-25T18:28:55.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:55.305+0000 7f23bfc25640 10 librados: watch_flush exit 2024-03-25T18:28:55.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:55.305+0000 7f23bfc25640 20 client.4415.objecter shutdown clearing up homeless session... 2024-03-25T18:28:55.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:55.305+0000 7f23bfc25640 10 client.4415.objecter successfully canceled tick 2024-03-25T18:28:55.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:55.305+0000 7f23bfc25640 1 -- 172.21.15.155:0/940728773 >> v1:172.21.15.167:6800/2919070503 conn(0x7f239c06df70 legacy=0x7f239c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:55.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:55.305+0000 7f23bfc25640 10 monclient: shutdown 2024-03-25T18:28:55.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:55.305+0000 7f23bfc25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:55.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:55.305+0000 7f23bfc25640 1 -- 172.21.15.155:0/940728773 >> v1:172.21.15.155:6790/0 conn(0x7f23b80a98f0 legacy=0x7f23b81daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:55.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:55.305+0000 7f23bfc25640 1 -- 172.21.15.155:0/940728773 shutdown_connections 2024-03-25T18:28:55.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:55.305+0000 7f23bfc25640 1 -- 172.21.15.155:0/940728773 >> 172.21.15.155:0/940728773 conn(0x7f23b80b7ea0 msgr2=0x7f23b8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:55.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:55.305+0000 7f23bfc25640 1 -- 172.21.15.155:0/940728773 shutdown_connections 2024-03-25T18:28:55.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:55.305+0000 7f23bfc25640 1 -- 172.21.15.155:0/940728773 wait complete. 2024-03-25T18:28:55.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:28:55.305+0000 7f23bfc25640 1 librados: shutdown 2024-03-25T18:28:55.318 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:01.113 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:04:34.717 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:01.158 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:04:34.736 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:01.160 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:04:34.764 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:01.160 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:04:34.739 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:01.162 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:04:34.766 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:01.176 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:04:34.781 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:01.196 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:04:34.801 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:01.210 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:04:34.789 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:01.219 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:04:34.824 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:02.125 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:02.239 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.238+0000 7f20ddf3b640 10 monclient: get_monmap_and_config 2024-03-25T18:29:02.239 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.238+0000 7f20ddf3b640 10 monclient: build_initial_monmap 2024-03-25T18:29:02.239 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.238+0000 7f20ddf3b640 10 monclient: monmap: 2024-03-25T18:29:02.239 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:29:02.239 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:29:02.239 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:29:02.239113+0000 2024-03-25T18:29:02.239 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:29:02.239113+0000 2024-03-25T18:29:02.239 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:02.239 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:29:02.239 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:29:02.240 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:29:02.240 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:29:02.240 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:29:02.241 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.238+0000 7f20ddf3b640 1 Processor -- start 2024-03-25T18:29:02.241 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.238+0000 7f20ddf3b640 1 -- start start 2024-03-25T18:29:02.241 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.238+0000 7f20ddf3b640 10 monclient: init 2024-03-25T18:29:02.241 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.239+0000 7f20ddf3b640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:02.241 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.239+0000 7f20ddf3b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:29:02.241 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.239+0000 7f20ddf3b640 10 monclient(hunting): picked mon.noname-b con 0x7f20d80b4d60 addr v1:172.21.15.155:6790/0 2024-03-25T18:29:02.241 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.239+0000 7f20ddf3b640 10 monclient(hunting): picked mon.noname-c con 0x7f20d80b1730 addr v1:172.21.15.167:6789/0 2024-03-25T18:29:02.241 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.239+0000 7f20ddf3b640 10 monclient(hunting): picked mon.noname-a con 0x7f20d814be90 addr v1:172.21.15.155:6789/0 2024-03-25T18:29:02.242 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.239+0000 7f20ddf3b640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20d80af940 con 0x7f20d814be90 2024-03-25T18:29:02.242 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.239+0000 7f20ddf3b640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20d80b0a70 con 0x7f20d80b1730 2024-03-25T18:29:02.242 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.239+0000 7f20ddf3b640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20d80b0d50 con 0x7f20d80b4d60 2024-03-25T18:29:02.242 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.239+0000 7f20ddf3b640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:02.242 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.239+0000 7f20ddf3b640 10 monclient(hunting): authenticate will time out at 7788.549420s 2024-03-25T18:29:02.242 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.240+0000 7f20d77fe640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f20d80b4d60 0x7f20d813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:50160/0 (socket says 172.21.15.155:50160) 2024-03-25T18:29:02.242 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.240+0000 7f20d77fe640 1 -- 172.21.15.155:0/4253688376 learned_addr learned my addr 172.21.15.155:0/4253688376 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:29:02.265 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.264+0000 7f20d67fc640 1 -- 172.21.15.155:0/4253688376 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 716442654 0 0) 0x7f20d80b0d50 con 0x7f20d80b4d60 2024-03-25T18:29:02.265 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.264+0000 7f20d67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:29:02.265 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.264+0000 7f20d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:02.265 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.264+0000 7f20d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:02.266 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.264+0000 7f20d67fc640 10 monclient(hunting): my global_id is 4418 2024-03-25T18:29:02.266 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.264+0000 7f20d67fc640 1 -- 172.21.15.155:0/4253688376 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20c4003520 con 0x7f20d80b4d60 2024-03-25T18:29:02.270 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.269+0000 7f20d67fc640 1 -- 172.21.15.155:0/4253688376 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 576525693 0 0) 0x7f20d80b0a70 con 0x7f20d80b1730 2024-03-25T18:29:02.270 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.269+0000 7f20d67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:29:02.271 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.269+0000 7f20d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:02.271 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.269+0000 7f20d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:02.271 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.269+0000 7f20d67fc640 10 monclient(hunting): my global_id is 4408 2024-03-25T18:29:02.271 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.269+0000 7f20d67fc640 1 -- 172.21.15.155:0/4253688376 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20d80b0d50 con 0x7f20d80b1730 2024-03-25T18:29:02.277 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.276+0000 7f20d67fc640 1 -- 172.21.15.155:0/4253688376 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 650893852 0 0) 0x7f20d80b0d50 con 0x7f20d80b1730 2024-03-25T18:29:02.277 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.276+0000 7f20d67fc640 1 -- 172.21.15.155:0/4253688376 >> v1:172.21.15.155:6790/0 conn(0x7f20d80b4d60 legacy=0x7f20d813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:02.277 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.276+0000 7f20d67fc640 1 -- 172.21.15.155:0/4253688376 >> v1:172.21.15.155:6789/0 conn(0x7f20d814be90 legacy=0x7f20d814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:02.277 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.276+0000 7f20d67fc640 10 monclient: _finish_hunting 0 2024-03-25T18:29:02.277 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.276+0000 7f20d67fc640 1 monclient: found mon.noname-c 2024-03-25T18:29:02.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.276+0000 7f20d67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:02.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.276+0000 7f20d67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.167:6789/0 2024-03-25T18:29:02.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.276+0000 7f20d67fc640 1 -- 172.21.15.155:0/4253688376 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20d80b1030 con 0x7f20d80b1730 2024-03-25T18:29:02.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.276+0000 7f20d67fc640 10 monclient: _finish_auth 0 2024-03-25T18:29:02.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.276+0000 7f20d67fc640 10 monclient: _check_auth_tickets 2024-03-25T18:29:02.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.276+0000 7f20d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:02.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.276+0000 7f20d67fc640 1 -- 172.21.15.155:0/4253688376 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f20c0001670 con 0x7f20d80b1730 2024-03-25T18:29:02.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.276+0000 7f20d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:02.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.276+0000 7f20d67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:29:02.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.276+0000 7f20d67fc640 10 monclient: dump: 2024-03-25T18:29:02.278 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:29:02.278 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:29:02.278 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:02.278 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:02.279 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:02.279 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:29:02.279 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:29:02.279 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:29:02.279 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:29:02.279 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:29:02.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.276+0000 7f20ddf3b640 5 monclient: authenticate success, global_id 4408 2024-03-25T18:29:02.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.276+0000 7f20ddf3b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:02.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.282+0000 7f20d67fc640 1 -- 172.21.15.155:0/4253688376 <== mon.1 v1:172.21.15.167:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f20c00017d0 con 0x7f20d80b1730 2024-03-25T18:29:02.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.282+0000 7f20d67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:02.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.282+0000 7f20d67fc640 1 -- 172.21.15.155:0/4253688376 <== mon.1 v1:172.21.15.167:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f20c0001ba0 con 0x7f20d80b1730 2024-03-25T18:29:02.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.282+0000 7f20d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:02.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.282+0000 7f20d67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:02.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.282+0000 7f20d67fc640 10 monclient: dump: 2024-03-25T18:29:02.284 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:29:02.284 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:29:02.284 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:02.284 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:02.284 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:02.284 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:29:02.284 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:29:02.284 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:29:02.284 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:29:02.284 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:29:02.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.282+0000 7f20ddf3b640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:02.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.282+0000 7f20ddf3b640 4 set_mon_vals no callback set 2024-03-25T18:29:02.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.283+0000 7f20ddf3b640 10 monclient: shutdown 2024-03-25T18:29:02.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.283+0000 7f20ddf3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:02.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.283+0000 7f20ddf3b640 1 -- 172.21.15.155:0/4253688376 >> v1:172.21.15.167:6789/0 conn(0x7f20d80b1730 legacy=0x7f20d80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:02.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.283+0000 7f20ddf3b640 1 -- 172.21.15.155:0/4253688376 shutdown_connections 2024-03-25T18:29:02.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.283+0000 7f20ddf3b640 1 -- 172.21.15.155:0/4253688376 >> 172.21.15.155:0/4253688376 conn(0x7f20d80617a0 msgr2=0x7f20d8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:02.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.283+0000 7f20ddf3b640 1 -- 172.21.15.155:0/4253688376 shutdown_connections 2024-03-25T18:29:02.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.283+0000 7f20ddf3b640 1 -- 172.21.15.155:0/4253688376 wait complete. 2024-03-25T18:29:02.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.284+0000 7f20ddf3b640 10 monclient: build_initial_monmap 2024-03-25T18:29:02.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.284+0000 7f20ddf3b640 10 monclient: monmap: 2024-03-25T18:29:02.287 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:29:02.287 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:02.287 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:29:02.287 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:29:02.288 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:02.288 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:29:02.288 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:29:02.288 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:29:02.288 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:29:02.288 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:29:02.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.284+0000 7f20ddf3b640 1 librados: starting msgr at 2024-03-25T18:29:02.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.284+0000 7f20ddf3b640 1 librados: starting objecter 2024-03-25T18:29:02.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.284+0000 7f20ddf3b640 1 Processor -- start 2024-03-25T18:29:02.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.284+0000 7f20ddf3b640 1 -- start start 2024-03-25T18:29:02.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.284+0000 7f20ddf3b640 1 librados: setting wanted keys 2024-03-25T18:29:02.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.284+0000 7f20ddf3b640 1 librados: calling monclient init 2024-03-25T18:29:02.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.284+0000 7f20ddf3b640 10 monclient: init 2024-03-25T18:29:02.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.285+0000 7f20ddf3b640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:02.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.285+0000 7f20ddf3b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:29:02.289 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.285+0000 7f20ddf3b640 10 monclient(hunting): picked mon.noname-a con 0x7f20d80b1730 addr v1:172.21.15.155:6789/0 2024-03-25T18:29:02.289 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.285+0000 7f20ddf3b640 10 monclient(hunting): picked mon.noname-c con 0x7f20d80b4d60 addr v1:172.21.15.155:6790/0 2024-03-25T18:29:02.289 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.285+0000 7f20ddf3b640 10 monclient(hunting): picked mon.noname-b con 0x7f20d814be90 addr v1:172.21.15.167:6789/0 2024-03-25T18:29:02.289 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.285+0000 7f20ddf3b640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20d81380f0 con 0x7f20d80b1730 2024-03-25T18:29:02.289 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.285+0000 7f20ddf3b640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20d8138340 con 0x7f20d814be90 2024-03-25T18:29:02.289 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.285+0000 7f20ddf3b640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20d8137470 con 0x7f20d80b4d60 2024-03-25T18:29:02.289 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.285+0000 7f20ddf3b640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:02.289 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.285+0000 7f20ddf3b640 10 monclient(hunting): authenticate will time out at 7788.595127s 2024-03-25T18:29:02.299 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.298+0000 7f20d7fff640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f20d814be90 0x7f20d81e25d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:40940/0 (socket says 172.21.15.155:40940) 2024-03-25T18:29:02.299 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.298+0000 7f20d7fff640 1 -- 172.21.15.155:0/3383382805 learned_addr learned my addr 172.21.15.155:0/3383382805 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:29:02.300 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.300+0000 7f20cffff640 10 client.?.objecter ms_handle_connect 0x7f20d814be90 2024-03-25T18:29:02.300 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.300+0000 7f20cffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:02.311 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.310+0000 7f20cffff640 10 client.?.objecter ms_handle_connect 0x7f20d80b4d60 2024-03-25T18:29:02.311 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.310+0000 7f20cffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:02.314 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.313+0000 7f20cffff640 1 -- 172.21.15.155:0/3383382805 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1277165200 0 0) 0x7f20d8138340 con 0x7f20d814be90 2024-03-25T18:29:02.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.314+0000 7f20cffff640 10 monclient(hunting): _negotiate 2024-03-25T18:29:02.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.314+0000 7f20cffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:02.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.314+0000 7f20cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:02.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.314+0000 7f20cffff640 10 monclient(hunting): my global_id is 4411 2024-03-25T18:29:02.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.314+0000 7f20cffff640 1 -- 172.21.15.155:0/3383382805 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20b8004eb0 con 0x7f20d814be90 2024-03-25T18:29:02.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.322+0000 7f20cffff640 1 -- 172.21.15.155:0/3383382805 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2832182060 0 0) 0x7f20d8137470 con 0x7f20d80b4d60 2024-03-25T18:29:02.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.322+0000 7f20cffff640 10 monclient(hunting): _negotiate 2024-03-25T18:29:02.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.322+0000 7f20cffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:02.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.322+0000 7f20cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:02.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.322+0000 7f20cffff640 10 monclient(hunting): my global_id is 4421 2024-03-25T18:29:02.324 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.322+0000 7f20cffff640 1 -- 172.21.15.155:0/3383382805 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20d8138340 con 0x7f20d80b4d60 2024-03-25T18:29:02.325 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.324+0000 7f20cffff640 1 -- 172.21.15.155:0/3383382805 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1107406458 0 0) 0x7f20b8004eb0 con 0x7f20d814be90 2024-03-25T18:29:02.325 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.324+0000 7f20cffff640 1 -- 172.21.15.155:0/3383382805 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f20d8137470 con 0x7f20d814be90 2024-03-25T18:29:02.325 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.324+0000 7f20cffff640 1 -- 172.21.15.155:0/3383382805 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f20c80018b0 con 0x7f20d814be90 2024-03-25T18:29:02.325 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.324+0000 7f20cffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:02.325 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.324+0000 7f20cffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:29:02.325 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.324+0000 7f20cffff640 10 monclient(hunting): dump: 2024-03-25T18:29:02.325 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:29:02.326 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:29:02.326 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:02.326 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:02.326 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:02.326 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:29:02.326 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:29:02.326 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:29:02.326 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:29:02.326 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:29:02.326 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.324+0000 7f20cffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:02.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.329+0000 7f20cffff640 10 client.?.objecter ms_handle_connect 0x7f20d80b1730 2024-03-25T18:29:02.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.329+0000 7f20cffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:02.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.339+0000 7f20cffff640 1 -- 172.21.15.155:0/3383382805 <== mon.1 v1:172.21.15.167:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4211174305 0 0) 0x7f20d8137470 con 0x7f20d814be90 2024-03-25T18:29:02.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.339+0000 7f20cffff640 1 -- 172.21.15.155:0/3383382805 >> v1:172.21.15.155:6790/0 conn(0x7f20d80b4d60 legacy=0x7f20d81392f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:02.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.339+0000 7f20cffff640 1 -- 172.21.15.155:0/3383382805 >> v1:172.21.15.155:6789/0 conn(0x7f20d80b1730 legacy=0x7f20d8136d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:02.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.339+0000 7f20cffff640 10 monclient: _finish_hunting 0 2024-03-25T18:29:02.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.339+0000 7f20cffff640 1 monclient: found mon.b 2024-03-25T18:29:02.343 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.339+0000 7f20cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:02.343 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.339+0000 7f20cffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:29:02.343 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.339+0000 7f20cffff640 1 -- 172.21.15.155:0/3383382805 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20d8137600 con 0x7f20d814be90 2024-03-25T18:29:02.343 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.339+0000 7f20cffff640 10 monclient: _finish_auth 0 2024-03-25T18:29:02.343 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.339+0000 7f20cffff640 10 monclient: _check_auth_tickets 2024-03-25T18:29:02.343 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.339+0000 7f20cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:02.344 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.339+0000 7f20ddf3b640 5 monclient: authenticate success, global_id 4411 2024-03-25T18:29:02.344 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.339+0000 7f20ddf3b640 10 monclient: _renew_subs 2024-03-25T18:29:02.344 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.339+0000 7f20ddf3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:29:02.344 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.339+0000 7f20ddf3b640 1 -- 172.21.15.155:0/3383382805 --> v1:172.21.15.167:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20d8135fc0 con 0x7f20d814be90 2024-03-25T18:29:02.345 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.339+0000 7f20ddf3b640 10 client.4411.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:02.345 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.339+0000 7f20ddf3b640 10 monclient: _renew_subs 2024-03-25T18:29:02.345 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.339+0000 7f20ddf3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:29:02.345 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.339+0000 7f20ddf3b640 1 -- 172.21.15.155:0/3383382805 --> v1:172.21.15.167:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f20d8139be0 con 0x7f20d814be90 2024-03-25T18:29:02.345 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.339+0000 7f20ddf3b640 1 librados: init done 2024-03-25T18:29:02.345 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.340+0000 7f20ddf3b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:02.346 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.340+0000 7f20ddf3b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:02.346 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.340+0000 7f20ddf3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:29:02.347 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.340+0000 7f20ddf3b640 1 -- 172.21.15.155:0/3383382805 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20d81401f0 con 0x7f20d814be90 2024-03-25T18:29:02.605 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.603+0000 7f20cffff640 1 -- 172.21.15.155:0/3383382805 <== mon.1 v1:172.21.15.167:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f20c80035f0 con 0x7f20d814be90 2024-03-25T18:29:02.605 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.603+0000 7f20cffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:02.605 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.603+0000 7f20cffff640 1 -- 172.21.15.155:0/3383382805 <== mon.1 v1:172.21.15.167:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f20c8003a00 con 0x7f20d814be90 2024-03-25T18:29:02.605 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.603+0000 7f20cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:02.605 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.603+0000 7f20cffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:02.605 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.603+0000 7f20cffff640 10 monclient: dump: 2024-03-25T18:29:02.605 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:29:02.605 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:29:02.605 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:02.605 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:02.605 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:02.606 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:29:02.606 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:29:02.606 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:29:02.606 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:29:02.606 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:29:02.606 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.603+0000 7f20d5ffb640 4 set_mon_vals no callback set 2024-03-25T18:29:02.606 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.604+0000 7f20cffff640 1 -- 172.21.15.155:0/3383382805 <== mon.1 v1:172.21.15.167:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7f20c8003c20 con 0x7f20d814be90 2024-03-25T18:29:02.607 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.605+0000 7f20d77fe640 10 monclient: get_auth_request con 0x7f20b806dd10 auth_method 0 2024-03-25T18:29:02.607 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.605+0000 7f20cffff640 10 client.4411.objecter ms_handle_connect 0x7f20b806dd10 2024-03-25T18:29:02.610 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.609+0000 7f20cffff640 1 -- 172.21.15.155:0/3383382805 <== mon.1 v1:172.21.15.167:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7f20c80546d0 con 0x7f20d814be90 2024-03-25T18:29:02.610 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.609+0000 7f20cffff640 10 client.4411.objecter ms_dispatch 0x7f20d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:02.610 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.609+0000 7f20cffff640 3 client.4411.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:02.610 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.609+0000 7f20cffff640 3 client.4411.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:02.610 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.609+0000 7f20cffff640 20 client.4411.objecter dump_active .. 0 homeless 2024-03-25T18:29:02.610 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.609+0000 7f20cffff640 1 -- 172.21.15.155:0/3383382805 <== mon.1 v1:172.21.15.167:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f20c80501e0 con 0x7f20d814be90 2024-03-25T18:29:02.610 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.609+0000 7f20cffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:02.611 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.609+0000 7f20cffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:02.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.857+0000 7f20ddf3b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:02.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.857+0000 7f20ddf3b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:02.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.857+0000 7f20ddf3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:29:02.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.857+0000 7f20ddf3b640 1 -- 172.21.15.155:0/3383382805 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f20d80627b0 con 0x7f20d814be90 2024-03-25T18:29:02.904 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.903+0000 7f20cffff640 1 -- 172.21.15.155:0/3383382805 <== mon.1 v1:172.21.15.167:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7f20c8052110 con 0x7f20d814be90 2024-03-25T18:29:02.904 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.903+0000 7f20cffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:02.905 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.903+0000 7f20cffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:02.905 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:29:02.905 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:02.907 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.906+0000 7f20ddf3b640 10 librados: watch_flush enter 2024-03-25T18:29:02.907 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.906+0000 7f20ddf3b640 10 librados: watch_flush exit 2024-03-25T18:29:02.907 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.906+0000 7f20ddf3b640 20 client.4411.objecter shutdown clearing up homeless session... 2024-03-25T18:29:02.907 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.906+0000 7f20ddf3b640 10 client.4411.objecter successfully canceled tick 2024-03-25T18:29:02.907 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.906+0000 7f20ddf3b640 1 -- 172.21.15.155:0/3383382805 >> v1:172.21.15.167:6800/2919070503 conn(0x7f20b806dd10 legacy=0x7f20b80701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:02.908 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.906+0000 7f20ddf3b640 10 monclient: shutdown 2024-03-25T18:29:02.908 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.906+0000 7f20ddf3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:02.908 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.906+0000 7f20ddf3b640 1 -- 172.21.15.155:0/3383382805 >> v1:172.21.15.167:6789/0 conn(0x7f20d814be90 legacy=0x7f20d81e25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:02.908 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.907+0000 7f20ddf3b640 1 -- 172.21.15.155:0/3383382805 shutdown_connections 2024-03-25T18:29:02.908 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.907+0000 7f20ddf3b640 1 -- 172.21.15.155:0/3383382805 >> 172.21.15.155:0/3383382805 conn(0x7f20d80617a0 msgr2=0x7f20d80b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:02.908 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.907+0000 7f20ddf3b640 1 -- 172.21.15.155:0/3383382805 shutdown_connections 2024-03-25T18:29:02.909 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.907+0000 7f20ddf3b640 1 -- 172.21.15.155:0/3383382805 wait complete. 2024-03-25T18:29:02.909 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:02.907+0000 7f20ddf3b640 1 librados: shutdown 2024-03-25T18:29:02.921 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:09.727 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:09.838 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.837+0000 7f7ce558c640 10 monclient: get_monmap_and_config 2024-03-25T18:29:09.838 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.837+0000 7f7ce558c640 10 monclient: build_initial_monmap 2024-03-25T18:29:09.838 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.837+0000 7f7ce558c640 10 monclient: monmap: 2024-03-25T18:29:09.838 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:29:09.838 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:29:09.838 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:29:09.838166+0000 2024-03-25T18:29:09.839 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:29:09.838166+0000 2024-03-25T18:29:09.839 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:09.839 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:29:09.839 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:29:09.839 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:29:09.839 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:29:09.839 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:29:09.840 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.837+0000 7f7ce558c640 1 Processor -- start 2024-03-25T18:29:09.840 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.837+0000 7f7ce558c640 1 -- start start 2024-03-25T18:29:09.840 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.837+0000 7f7ce558c640 10 monclient: init 2024-03-25T18:29:09.841 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.838+0000 7f7ce558c640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:09.841 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.838+0000 7f7ce558c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:29:09.841 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.838+0000 7f7ce558c640 10 monclient(hunting): picked mon.noname-c con 0x7f7ce006c380 addr v1:172.21.15.167:6789/0 2024-03-25T18:29:09.841 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.838+0000 7f7ce558c640 10 monclient(hunting): picked mon.noname-b con 0x7f7ce006d670 addr v1:172.21.15.155:6790/0 2024-03-25T18:29:09.841 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.838+0000 7f7ce558c640 10 monclient(hunting): picked mon.noname-a con 0x7f7ce00af8f0 addr v1:172.21.15.155:6789/0 2024-03-25T18:29:09.841 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.838+0000 7f7ce558c640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ce006c9a0 con 0x7f7ce00af8f0 2024-03-25T18:29:09.841 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.838+0000 7f7ce558c640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ce00ada90 con 0x7f7ce006c380 2024-03-25T18:29:09.841 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.838+0000 7f7ce558c640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ce00b0d10 con 0x7f7ce006d670 2024-03-25T18:29:09.841 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.838+0000 7f7ce558c640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:09.841 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.838+0000 7f7ce558c640 10 monclient(hunting): authenticate will time out at 7796.148393s 2024-03-25T18:29:09.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.852+0000 7f7cde7fc640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f7ce006d670 0x7f7ce00ae6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:41932/0 (socket says 172.21.15.155:41932) 2024-03-25T18:29:09.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.852+0000 7f7cde7fc640 1 -- 172.21.15.155:0/4230396636 learned_addr learned my addr 172.21.15.155:0/4230396636 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:29:09.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.863+0000 7f7cddffb640 1 -- 172.21.15.155:0/4230396636 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3186368013 0 0) 0x7f7ce00b0d10 con 0x7f7ce006d670 2024-03-25T18:29:09.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.863+0000 7f7cddffb640 10 monclient(hunting): _negotiate 2024-03-25T18:29:09.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.863+0000 7f7cddffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:09.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.863+0000 7f7cddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:09.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.863+0000 7f7cddffb640 10 monclient(hunting): my global_id is 4424 2024-03-25T18:29:09.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.863+0000 7f7cddffb640 1 -- 172.21.15.155:0/4230396636 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cc8003520 con 0x7f7ce006d670 2024-03-25T18:29:09.872 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.871+0000 7f7cddffb640 1 -- 172.21.15.155:0/4230396636 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2983734985 0 0) 0x7f7ce006c9a0 con 0x7f7ce00af8f0 2024-03-25T18:29:09.872 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.871+0000 7f7cddffb640 10 monclient(hunting): _negotiate 2024-03-25T18:29:09.872 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.871+0000 7f7cddffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:09.872 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.871+0000 7f7cddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:09.872 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.871+0000 7f7cddffb640 10 monclient(hunting): my global_id is 4419 2024-03-25T18:29:09.872 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.871+0000 7f7cddffb640 1 -- 172.21.15.155:0/4230396636 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ce00b0d10 con 0x7f7ce00af8f0 2024-03-25T18:29:09.878 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.877+0000 7f7cddffb640 1 -- 172.21.15.155:0/4230396636 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3408294628 0 0) 0x7f7ce00ada90 con 0x7f7ce006c380 2024-03-25T18:29:09.878 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.877+0000 7f7cddffb640 10 monclient(hunting): _negotiate 2024-03-25T18:29:09.878 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.877+0000 7f7cddffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:09.879 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.877+0000 7f7cddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:09.879 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.877+0000 7f7cddffb640 10 monclient(hunting): my global_id is 4414 2024-03-25T18:29:09.879 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.877+0000 7f7cddffb640 1 -- 172.21.15.155:0/4230396636 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ce006c9a0 con 0x7f7ce006c380 2024-03-25T18:29:09.892 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.891+0000 7f7cddffb640 1 -- 172.21.15.155:0/4230396636 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1033837658 0 0) 0x7f7ce00b0d10 con 0x7f7ce00af8f0 2024-03-25T18:29:09.892 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.891+0000 7f7cddffb640 1 -- 172.21.15.155:0/4230396636 >> v1:172.21.15.155:6790/0 conn(0x7f7ce006d670 legacy=0x7f7ce00ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:09.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.891+0000 7f7cddffb640 1 -- 172.21.15.155:0/4230396636 >> v1:172.21.15.167:6789/0 conn(0x7f7ce006c380 legacy=0x7f7ce00b0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:09.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.891+0000 7f7cddffb640 10 monclient: _finish_hunting 0 2024-03-25T18:29:09.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.891+0000 7f7cddffb640 1 monclient: found mon.noname-a 2024-03-25T18:29:09.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.891+0000 7f7cddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:09.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.891+0000 7f7cddffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-25T18:29:09.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.891+0000 7f7cddffb640 1 -- 172.21.15.155:0/4230396636 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ce00add00 con 0x7f7ce00af8f0 2024-03-25T18:29:09.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.891+0000 7f7cddffb640 10 monclient: _finish_auth 0 2024-03-25T18:29:09.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.891+0000 7f7cddffb640 10 monclient: _check_auth_tickets 2024-03-25T18:29:09.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.891+0000 7f7cddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:09.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.891+0000 7f7cddffb640 1 -- 172.21.15.155:0/4230396636 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f7cd80019f0 con 0x7f7ce00af8f0 2024-03-25T18:29:09.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.891+0000 7f7cddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:09.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.891+0000 7f7cddffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:29:09.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.891+0000 7f7cddffb640 10 monclient: dump: 2024-03-25T18:29:09.894 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:29:09.894 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:29:09.894 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:09.894 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:09.894 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:09.894 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:29:09.894 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:29:09.894 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:29:09.894 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:29:09.894 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:29:09.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.891+0000 7f7ce558c640 5 monclient: authenticate success, global_id 4419 2024-03-25T18:29:09.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.891+0000 7f7ce558c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:09.919 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.918+0000 7f7cddffb640 1 -- 172.21.15.155:0/4230396636 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7cd8001b50 con 0x7f7ce00af8f0 2024-03-25T18:29:09.919 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.918+0000 7f7cddffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:09.919 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.918+0000 7f7cddffb640 1 -- 172.21.15.155:0/4230396636 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f7cd8001ee0 con 0x7f7ce00af8f0 2024-03-25T18:29:09.919 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.918+0000 7f7cddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:09.919 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.918+0000 7f7cddffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:09.919 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.918+0000 7f7cddffb640 10 monclient: dump: 2024-03-25T18:29:09.919 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:29:09.919 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:29:09.919 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:09.919 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:09.919 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:09.919 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:29:09.920 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:29:09.920 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:29:09.920 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:29:09.920 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:29:09.920 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.918+0000 7f7ce558c640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:09.920 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.918+0000 7f7ce558c640 4 set_mon_vals no callback set 2024-03-25T18:29:09.922 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.918+0000 7f7ce558c640 10 monclient: shutdown 2024-03-25T18:29:09.922 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.918+0000 7f7ce558c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:09.922 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.918+0000 7f7ce558c640 1 -- 172.21.15.155:0/4230396636 >> v1:172.21.15.155:6789/0 conn(0x7f7ce00af8f0 legacy=0x7f7ce00aedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:09.922 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.918+0000 7f7ce558c640 1 -- 172.21.15.155:0/4230396636 shutdown_connections 2024-03-25T18:29:09.923 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.919+0000 7f7ce558c640 1 -- 172.21.15.155:0/4230396636 >> 172.21.15.155:0/4230396636 conn(0x7f7ce00b5c00 msgr2=0x7f7ce00b5fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:09.923 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.919+0000 7f7ce558c640 1 -- 172.21.15.155:0/4230396636 shutdown_connections 2024-03-25T18:29:09.923 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.919+0000 7f7ce558c640 1 -- 172.21.15.155:0/4230396636 wait complete. 2024-03-25T18:29:09.923 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.919+0000 7f7ce558c640 10 monclient: build_initial_monmap 2024-03-25T18:29:09.923 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.919+0000 7f7ce558c640 10 monclient: monmap: 2024-03-25T18:29:09.923 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:29:09.923 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:09.923 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:29:09.923 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:29:09.923 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:09.923 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:29:09.923 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:29:09.923 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:29:09.923 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:29:09.924 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:29:09.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.920+0000 7f7ce558c640 1 librados: starting msgr at 2024-03-25T18:29:09.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.920+0000 7f7ce558c640 1 librados: starting objecter 2024-03-25T18:29:09.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.920+0000 7f7ce558c640 1 Processor -- start 2024-03-25T18:29:09.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.920+0000 7f7ce558c640 1 -- start start 2024-03-25T18:29:09.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.920+0000 7f7ce558c640 1 librados: setting wanted keys 2024-03-25T18:29:09.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.920+0000 7f7ce558c640 1 librados: calling monclient init 2024-03-25T18:29:09.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.920+0000 7f7ce558c640 10 monclient: init 2024-03-25T18:29:09.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.920+0000 7f7ce558c640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:09.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.920+0000 7f7ce558c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:29:09.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.920+0000 7f7ce558c640 10 monclient(hunting): picked mon.noname-b con 0x7f7ce006c380 addr v1:172.21.15.167:6789/0 2024-03-25T18:29:09.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.920+0000 7f7ce558c640 10 monclient(hunting): picked mon.noname-a con 0x7f7ce006d670 addr v1:172.21.15.155:6789/0 2024-03-25T18:29:09.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.920+0000 7f7ce558c640 10 monclient(hunting): picked mon.noname-c con 0x7f7ce00af8f0 addr v1:172.21.15.155:6790/0 2024-03-25T18:29:09.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.920+0000 7f7ce558c640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ce013f9b0 con 0x7f7ce006d670 2024-03-25T18:29:09.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.920+0000 7f7ce558c640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ce013fc20 con 0x7f7ce006c380 2024-03-25T18:29:09.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.920+0000 7f7ce558c640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ce013ff00 con 0x7f7ce00af8f0 2024-03-25T18:29:09.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.920+0000 7f7ce558c640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:09.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.920+0000 7f7ce558c640 10 monclient(hunting): authenticate will time out at 7796.230683s 2024-03-25T18:29:09.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.924+0000 7f7cdeffd640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f7ce006c380 0x7f7ce0148300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:43746/0 (socket says 172.21.15.155:43746) 2024-03-25T18:29:09.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.924+0000 7f7cdeffd640 1 -- 172.21.15.155:0/4140231345 learned_addr learned my addr 172.21.15.155:0/4140231345 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:29:09.929 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.929+0000 7f7cc77fe640 10 client.?.objecter ms_handle_connect 0x7f7ce006c380 2024-03-25T18:29:09.929 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.929+0000 7f7cc77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:09.936 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.935+0000 7f7cc77fe640 10 client.?.objecter ms_handle_connect 0x7f7ce006d670 2024-03-25T18:29:09.936 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.935+0000 7f7cc77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:09.937 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.936+0000 7f7cc77fe640 10 client.?.objecter ms_handle_connect 0x7f7ce00af8f0 2024-03-25T18:29:09.937 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.936+0000 7f7cc77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:09.942 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.941+0000 7f7cc77fe640 1 -- 172.21.15.155:0/4140231345 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 235457179 0 0) 0x7f7ce013fc20 con 0x7f7ce006c380 2024-03-25T18:29:09.942 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.941+0000 7f7cc77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:29:09.942 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.941+0000 7f7cc77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:09.942 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.941+0000 7f7cc77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:09.942 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.941+0000 7f7cc77fe640 10 monclient(hunting): my global_id is 4417 2024-03-25T18:29:09.942 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.941+0000 7f7cc77fe640 1 -- 172.21.15.155:0/4140231345 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ccc004eb0 con 0x7f7ce006c380 2024-03-25T18:29:09.946 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.945+0000 7f7cc77fe640 1 -- 172.21.15.155:0/4140231345 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 873236089 0 0) 0x7f7ce013ff00 con 0x7f7ce00af8f0 2024-03-25T18:29:09.946 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.945+0000 7f7cc77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:29:09.946 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.945+0000 7f7cc77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:09.947 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.945+0000 7f7cc77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:09.947 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.945+0000 7f7cc77fe640 10 monclient(hunting): my global_id is 4427 2024-03-25T18:29:09.947 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.945+0000 7f7cc77fe640 1 -- 172.21.15.155:0/4140231345 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ce013fc20 con 0x7f7ce00af8f0 2024-03-25T18:29:09.952 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.951+0000 7f7cc77fe640 1 -- 172.21.15.155:0/4140231345 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3912798929 0 0) 0x7f7ce013f9b0 con 0x7f7ce006d670 2024-03-25T18:29:09.952 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.951+0000 7f7cc77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:29:09.952 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.951+0000 7f7cc77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:09.952 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.951+0000 7f7cc77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:09.952 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.951+0000 7f7cc77fe640 10 monclient(hunting): my global_id is 4422 2024-03-25T18:29:09.952 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.951+0000 7f7cc77fe640 1 -- 172.21.15.155:0/4140231345 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ce013ff00 con 0x7f7ce006d670 2024-03-25T18:29:09.955 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.953+0000 7f7cc77fe640 1 -- 172.21.15.155:0/4140231345 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 309215486 0 0) 0x7f7ccc004eb0 con 0x7f7ce006c380 2024-03-25T18:29:09.955 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.953+0000 7f7cc77fe640 1 -- 172.21.15.155:0/4140231345 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ce013f9b0 con 0x7f7ce006c380 2024-03-25T18:29:09.955 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.954+0000 7f7cc77fe640 1 -- 172.21.15.155:0/4140231345 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f7cd0001a10 con 0x7f7ce006c380 2024-03-25T18:29:09.955 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.954+0000 7f7cc77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:09.955 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.954+0000 7f7cc77fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:29:09.955 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.954+0000 7f7cc77fe640 10 monclient(hunting): dump: 2024-03-25T18:29:09.955 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:29:09.955 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:29:09.955 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:09.955 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:09.955 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:09.955 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:29:09.955 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:29:09.956 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:29:09.956 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:29:09.956 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:29:09.956 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.954+0000 7f7cc77fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:09.956 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.955+0000 7f7cc77fe640 1 -- 172.21.15.155:0/4140231345 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 593227253 0 0) 0x7f7ce013fc20 con 0x7f7ce00af8f0 2024-03-25T18:29:09.956 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.955+0000 7f7cc77fe640 1 -- 172.21.15.155:0/4140231345 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ccc004eb0 con 0x7f7ce00af8f0 2024-03-25T18:29:09.956 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.955+0000 7f7cc77fe640 1 -- 172.21.15.155:0/4140231345 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f7cd8001e60 con 0x7f7ce00af8f0 2024-03-25T18:29:09.956 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.955+0000 7f7cc77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:09.956 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.955+0000 7f7cc77fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:09.956 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.955+0000 7f7cc77fe640 10 monclient(hunting): dump: 2024-03-25T18:29:09.957 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:29:09.957 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:29:09.957 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:09.957 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:09.957 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:09.957 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:29:09.957 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:29:09.957 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:29:09.957 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:29:09.957 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:29:09.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.971+0000 7f7cc77fe640 1 -- 172.21.15.155:0/4140231345 <== mon.1 v1:172.21.15.167:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2164406464 0 0) 0x7f7ce013f9b0 con 0x7f7ce006c380 2024-03-25T18:29:09.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.971+0000 7f7cc77fe640 1 -- 172.21.15.155:0/4140231345 >> v1:172.21.15.155:6790/0 conn(0x7f7ce00af8f0 legacy=0x7f7ce01e2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:09.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.971+0000 7f7cc77fe640 1 -- 172.21.15.155:0/4140231345 >> v1:172.21.15.155:6789/0 conn(0x7f7ce006d670 legacy=0x7f7ce0140600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:09.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.971+0000 7f7cc77fe640 10 monclient: _finish_hunting 0 2024-03-25T18:29:09.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.971+0000 7f7cc77fe640 1 monclient: found mon.b 2024-03-25T18:29:09.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.971+0000 7f7cc77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:09.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.971+0000 7f7cc77fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:29:09.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.971+0000 7f7cc77fe640 1 -- 172.21.15.155:0/4140231345 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ce01401e0 con 0x7f7ce006c380 2024-03-25T18:29:09.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.971+0000 7f7cc77fe640 10 monclient: _finish_auth 0 2024-03-25T18:29:09.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.971+0000 7f7cc77fe640 10 monclient: _check_auth_tickets 2024-03-25T18:29:09.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.971+0000 7f7cc77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:09.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.971+0000 7f7ce558c640 5 monclient: authenticate success, global_id 4417 2024-03-25T18:29:09.974 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.971+0000 7f7ce558c640 10 monclient: _renew_subs 2024-03-25T18:29:09.974 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.971+0000 7f7ce558c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:29:09.974 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.971+0000 7f7ce558c640 1 -- 172.21.15.155:0/4140231345 --> v1:172.21.15.167:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ce0146d50 con 0x7f7ce006c380 2024-03-25T18:29:09.974 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.971+0000 7f7ce558c640 10 client.4417.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:09.974 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.971+0000 7f7ce558c640 10 monclient: _renew_subs 2024-03-25T18:29:09.974 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.971+0000 7f7ce558c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:29:09.974 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.971+0000 7f7ce558c640 1 -- 172.21.15.155:0/4140231345 --> v1:172.21.15.167:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ce013ed80 con 0x7f7ce006c380 2024-03-25T18:29:09.974 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.971+0000 7f7ce558c640 1 librados: init done 2024-03-25T18:29:09.975 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.972+0000 7f7ce558c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:09.975 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.972+0000 7f7ce558c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:09.975 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.972+0000 7f7ce558c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:29:09.975 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:09.972+0000 7f7ce558c640 1 -- 172.21.15.155:0/4140231345 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ce0146b90 con 0x7f7ce006c380 2024-03-25T18:29:10.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:10.258+0000 7f7cc77fe640 1 -- 172.21.15.155:0/4140231345 <== mon.1 v1:172.21.15.167:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7cd0001c70 con 0x7f7ce006c380 2024-03-25T18:29:10.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:10.258+0000 7f7cc77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:10.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:10.258+0000 7f7cc77fe640 1 -- 172.21.15.155:0/4140231345 <== mon.1 v1:172.21.15.167:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f7cd0003a60 con 0x7f7ce006c380 2024-03-25T18:29:10.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:10.258+0000 7f7cc77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:10.260 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:10.258+0000 7f7cc77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:10.260 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:10.258+0000 7f7cc77fe640 10 monclient: dump: 2024-03-25T18:29:10.260 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:29:10.260 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:29:10.260 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:10.260 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:10.260 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:10.260 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:29:10.260 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:29:10.260 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:29:10.260 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:29:10.260 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:29:10.260 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:10.258+0000 7f7cdd7fa640 4 set_mon_vals no callback set 2024-03-25T18:29:10.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:10.259+0000 7f7cc77fe640 1 -- 172.21.15.155:0/4140231345 <== mon.1 v1:172.21.15.167:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7f7cd0003c80 con 0x7f7ce006c380 2024-03-25T18:29:10.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:10.259+0000 7f7cde7fc640 10 monclient: get_auth_request con 0x7f7ccc06df90 auth_method 0 2024-03-25T18:29:10.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:10.260+0000 7f7cc77fe640 10 client.4417.objecter ms_handle_connect 0x7f7ccc06df90 2024-03-25T18:29:10.266 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:10.265+0000 7f7cc77fe640 1 -- 172.21.15.155:0/4140231345 <== mon.1 v1:172.21.15.167:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7f7cd0054820 con 0x7f7ce006c380 2024-03-25T18:29:10.266 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:10.265+0000 7f7cc77fe640 10 client.4417.objecter ms_dispatch 0x7f7ce0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:10.266 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:10.265+0000 7f7cc77fe640 3 client.4417.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:10.266 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:10.265+0000 7f7cc77fe640 3 client.4417.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:10.266 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:10.265+0000 7f7cc77fe640 20 client.4417.objecter dump_active .. 0 homeless 2024-03-25T18:29:10.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:10.266+0000 7f7cc77fe640 1 -- 172.21.15.155:0/4140231345 <== mon.1 v1:172.21.15.167:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7cd0050330 con 0x7f7ce006c380 2024-03-25T18:29:10.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:10.266+0000 7f7cc77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:10.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:10.266+0000 7f7cc77fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:10.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:10.523+0000 7f7ce558c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:10.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:10.523+0000 7f7ce558c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:10.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:10.523+0000 7f7ce558c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:29:10.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:10.523+0000 7f7ce558c640 1 -- 172.21.15.155:0/4140231345 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7ce00627b0 con 0x7f7ce006c380 2024-03-25T18:29:10.554 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:10.552+0000 7f7cc77fe640 1 -- 172.21.15.155:0/4140231345 <== mon.1 v1:172.21.15.167:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7f7cd0052260 con 0x7f7ce006c380 2024-03-25T18:29:10.554 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:10.552+0000 7f7cc77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:10.554 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:10.552+0000 7f7cc77fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:10.554 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:29:10.554 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:10.556 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:10.555+0000 7f7ce558c640 10 librados: watch_flush enter 2024-03-25T18:29:10.556 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:10.555+0000 7f7ce558c640 10 librados: watch_flush exit 2024-03-25T18:29:10.557 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:10.555+0000 7f7ce558c640 20 client.4417.objecter shutdown clearing up homeless session... 2024-03-25T18:29:10.557 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:10.555+0000 7f7ce558c640 10 client.4417.objecter successfully canceled tick 2024-03-25T18:29:10.557 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:10.555+0000 7f7ce558c640 1 -- 172.21.15.155:0/4140231345 >> v1:172.21.15.167:6800/2919070503 conn(0x7f7ccc06df90 legacy=0x7f7ccc070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:10.557 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:10.555+0000 7f7ce558c640 10 monclient: shutdown 2024-03-25T18:29:10.557 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:10.555+0000 7f7ce558c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:10.557 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:10.555+0000 7f7ce558c640 1 -- 172.21.15.155:0/4140231345 >> v1:172.21.15.167:6789/0 conn(0x7f7ce006c380 legacy=0x7f7ce0148300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:10.557 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:10.555+0000 7f7ce558c640 1 -- 172.21.15.155:0/4140231345 shutdown_connections 2024-03-25T18:29:10.557 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:10.555+0000 7f7ce558c640 1 -- 172.21.15.155:0/4140231345 >> 172.21.15.155:0/4140231345 conn(0x7f7ce00b5c00 msgr2=0x7f7ce0144e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:10.557 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:10.555+0000 7f7ce558c640 1 -- 172.21.15.155:0/4140231345 shutdown_connections 2024-03-25T18:29:10.557 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:10.555+0000 7f7ce558c640 1 -- 172.21.15.155:0/4140231345 wait complete. 2024-03-25T18:29:10.557 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:10.555+0000 7f7ce558c640 1 librados: shutdown 2024-03-25T18:29:10.569 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:17.375 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:17.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.495+0000 7f249c33b640 10 monclient: get_monmap_and_config 2024-03-25T18:29:17.497 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.495+0000 7f249c33b640 10 monclient: build_initial_monmap 2024-03-25T18:29:17.497 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.495+0000 7f249c33b640 10 monclient: monmap: 2024-03-25T18:29:17.497 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:29:17.497 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:29:17.497 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:29:17.496767+0000 2024-03-25T18:29:17.497 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:29:17.496767+0000 2024-03-25T18:29:17.497 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:17.497 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:29:17.497 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:29:17.497 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:29:17.497 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:29:17.497 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:29:17.498 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.496+0000 7f249c33b640 1 Processor -- start 2024-03-25T18:29:17.499 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.496+0000 7f249c33b640 1 -- start start 2024-03-25T18:29:17.499 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.496+0000 7f249c33b640 10 monclient: init 2024-03-25T18:29:17.499 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.497+0000 7f249c33b640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:17.499 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.497+0000 7f249c33b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:29:17.499 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.497+0000 7f249c33b640 10 monclient(hunting): picked mon.noname-c con 0x7f24940a98f0 addr v1:172.21.15.167:6789/0 2024-03-25T18:29:17.499 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.497+0000 7f249c33b640 10 monclient(hunting): picked mon.noname-b con 0x7f249413f710 addr v1:172.21.15.155:6790/0 2024-03-25T18:29:17.499 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.497+0000 7f249c33b640 10 monclient(hunting): picked mon.noname-a con 0x7f24941430c0 addr v1:172.21.15.155:6789/0 2024-03-25T18:29:17.499 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.497+0000 7f249c33b640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2494142520 con 0x7f24941430c0 2024-03-25T18:29:17.499 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.497+0000 7f249c33b640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24940b28a0 con 0x7f24940a98f0 2024-03-25T18:29:17.499 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.497+0000 7f249c33b640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24940b2a30 con 0x7f249413f710 2024-03-25T18:29:17.499 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.497+0000 7f249c33b640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:17.499 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.497+0000 7f249c33b640 10 monclient(hunting): authenticate will time out at 7803.807141s 2024-03-25T18:29:17.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.508+0000 7f24998d6640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f249413f710 0x7f24941429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:36244/0 (socket says 172.21.15.155:36244) 2024-03-25T18:29:17.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.508+0000 7f24998d6640 1 -- 172.21.15.155:0/277864808 learned_addr learned my addr 172.21.15.155:0/277864808 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:29:17.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.518+0000 7f24990d5640 1 -- 172.21.15.155:0/277864808 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 122733029 0 0) 0x7f24940b2a30 con 0x7f249413f710 2024-03-25T18:29:17.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.518+0000 7f24990d5640 10 monclient(hunting): _negotiate 2024-03-25T18:29:17.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.518+0000 7f24990d5640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:17.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.518+0000 7f24990d5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:17.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.518+0000 7f24990d5640 10 monclient(hunting): my global_id is 4430 2024-03-25T18:29:17.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.518+0000 7f24990d5640 1 -- 172.21.15.155:0/277864808 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2480003520 con 0x7f249413f710 2024-03-25T18:29:17.526 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.525+0000 7f24990d5640 1 -- 172.21.15.155:0/277864808 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3759218999 0 0) 0x7f2480003520 con 0x7f249413f710 2024-03-25T18:29:17.526 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.525+0000 7f24990d5640 1 -- 172.21.15.155:0/277864808 >> v1:172.21.15.167:6789/0 conn(0x7f24940a98f0 legacy=0x7f24940a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:17.526 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.525+0000 7f24990d5640 1 -- 172.21.15.155:0/277864808 >> v1:172.21.15.155:6789/0 conn(0x7f24941430c0 legacy=0x7f249414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:17.526 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.525+0000 7f24990d5640 10 monclient: _finish_hunting 0 2024-03-25T18:29:17.526 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.525+0000 7f24990d5640 1 monclient: found mon.noname-b 2024-03-25T18:29:17.526 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.525+0000 7f24990d5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:17.526 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.525+0000 7f24990d5640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-25T18:29:17.527 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.525+0000 7f24990d5640 1 -- 172.21.15.155:0/277864808 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24940b44c0 con 0x7f249413f710 2024-03-25T18:29:17.527 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.525+0000 7f24990d5640 10 monclient: _finish_auth 0 2024-03-25T18:29:17.527 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.525+0000 7f24990d5640 10 monclient: _check_auth_tickets 2024-03-25T18:29:17.527 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.525+0000 7f24990d5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:17.527 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.525+0000 7f24990d5640 1 -- 172.21.15.155:0/277864808 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f2484001690 con 0x7f249413f710 2024-03-25T18:29:17.527 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.525+0000 7f24990d5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:17.527 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.525+0000 7f24990d5640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:29:17.527 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.525+0000 7f24990d5640 10 monclient: dump: 2024-03-25T18:29:17.527 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:29:17.527 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:29:17.527 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:17.527 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:17.527 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:17.527 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:29:17.527 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:29:17.528 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:29:17.528 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:29:17.528 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:29:17.528 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.525+0000 7f249c33b640 5 monclient: authenticate success, global_id 4430 2024-03-25T18:29:17.528 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.525+0000 7f249c33b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:17.541 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.540+0000 7f24990d5640 1 -- 172.21.15.155:0/277864808 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24840017f0 con 0x7f249413f710 2024-03-25T18:29:17.542 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.540+0000 7f24990d5640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:17.542 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.540+0000 7f24990d5640 1 -- 172.21.15.155:0/277864808 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f2484001c00 con 0x7f249413f710 2024-03-25T18:29:17.542 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.540+0000 7f24990d5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:17.542 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.540+0000 7f24990d5640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:17.542 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.540+0000 7f24990d5640 10 monclient: dump: 2024-03-25T18:29:17.542 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:29:17.542 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:29:17.542 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:17.542 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:17.542 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:17.542 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:29:17.543 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:29:17.543 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:29:17.543 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:29:17.543 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:29:17.543 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.540+0000 7f249c33b640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:17.543 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.540+0000 7f249c33b640 4 set_mon_vals no callback set 2024-03-25T18:29:17.545 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.541+0000 7f249c33b640 10 monclient: shutdown 2024-03-25T18:29:17.545 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.541+0000 7f249c33b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:17.545 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.541+0000 7f249c33b640 1 -- 172.21.15.155:0/277864808 >> v1:172.21.15.155:6790/0 conn(0x7f249413f710 legacy=0x7f24941429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:17.545 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.541+0000 7f249c33b640 1 -- 172.21.15.155:0/277864808 shutdown_connections 2024-03-25T18:29:17.545 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.541+0000 7f249c33b640 1 -- 172.21.15.155:0/277864808 >> 172.21.15.155:0/277864808 conn(0x7f24940b7ea0 msgr2=0x7f24940b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:17.545 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.541+0000 7f249c33b640 1 -- 172.21.15.155:0/277864808 shutdown_connections 2024-03-25T18:29:17.545 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.541+0000 7f249c33b640 1 -- 172.21.15.155:0/277864808 wait complete. 2024-03-25T18:29:17.545 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.542+0000 7f249c33b640 10 monclient: build_initial_monmap 2024-03-25T18:29:17.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.542+0000 7f249c33b640 10 monclient: monmap: 2024-03-25T18:29:17.546 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:29:17.546 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:17.546 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:29:17.546 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:29:17.546 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:17.546 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:29:17.546 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:29:17.546 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:29:17.546 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:29:17.546 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:29:17.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.542+0000 7f249c33b640 1 librados: starting msgr at 2024-03-25T18:29:17.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.542+0000 7f249c33b640 1 librados: starting objecter 2024-03-25T18:29:17.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.543+0000 7f249c33b640 1 Processor -- start 2024-03-25T18:29:17.547 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.543+0000 7f249c33b640 1 -- start start 2024-03-25T18:29:17.547 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.543+0000 7f249c33b640 1 librados: setting wanted keys 2024-03-25T18:29:17.547 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.543+0000 7f249c33b640 1 librados: calling monclient init 2024-03-25T18:29:17.547 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.543+0000 7f249c33b640 10 monclient: init 2024-03-25T18:29:17.547 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.543+0000 7f249c33b640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:17.547 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.543+0000 7f249c33b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:29:17.547 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.543+0000 7f249c33b640 10 monclient(hunting): picked mon.noname-b con 0x7f24940a98f0 addr v1:172.21.15.167:6789/0 2024-03-25T18:29:17.547 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.543+0000 7f249c33b640 10 monclient(hunting): picked mon.noname-a con 0x7f249413f710 addr v1:172.21.15.155:6789/0 2024-03-25T18:29:17.547 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.543+0000 7f249c33b640 10 monclient(hunting): picked mon.noname-c con 0x7f24941430c0 addr v1:172.21.15.155:6790/0 2024-03-25T18:29:17.547 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.543+0000 7f249c33b640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24941398b0 con 0x7f249413f710 2024-03-25T18:29:17.547 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.543+0000 7f249c33b640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24940711a0 con 0x7f24940a98f0 2024-03-25T18:29:17.547 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.544+0000 7f249c33b640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f249406f730 con 0x7f24941430c0 2024-03-25T18:29:17.547 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.544+0000 7f249c33b640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:17.547 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.544+0000 7f249c33b640 10 monclient(hunting): authenticate will time out at 7803.853749s 2024-03-25T18:29:17.558 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.557+0000 7f24998d6640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f249413f710 0x7f24941e32b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:58304/0 (socket says 172.21.15.155:58304) 2024-03-25T18:29:17.558 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.557+0000 7f24998d6640 1 -- 172.21.15.155:0/2666893628 learned_addr learned my addr 172.21.15.155:0/2666893628 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:29:17.559 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.559+0000 7f248affd640 10 client.?.objecter ms_handle_connect 0x7f249413f710 2024-03-25T18:29:17.560 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.559+0000 7f248affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:17.564 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.563+0000 7f248affd640 10 client.?.objecter ms_handle_connect 0x7f24941430c0 2024-03-25T18:29:17.564 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.563+0000 7f248affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:17.566 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.565+0000 7f248affd640 10 client.?.objecter ms_handle_connect 0x7f24940a98f0 2024-03-25T18:29:17.566 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.565+0000 7f248affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:17.570 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.569+0000 7f248affd640 1 -- 172.21.15.155:0/2666893628 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 99068790 0 0) 0x7f24941398b0 con 0x7f249413f710 2024-03-25T18:29:17.570 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.569+0000 7f248affd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:17.570 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.569+0000 7f248affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:17.570 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.569+0000 7f248affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:17.570 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.569+0000 7f248affd640 10 monclient(hunting): my global_id is 4428 2024-03-25T18:29:17.570 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.569+0000 7f248affd640 1 -- 172.21.15.155:0/2666893628 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2478004eb0 con 0x7f249413f710 2024-03-25T18:29:17.576 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.575+0000 7f248affd640 1 -- 172.21.15.155:0/2666893628 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2571538642 0 0) 0x7f24940711a0 con 0x7f24940a98f0 2024-03-25T18:29:17.576 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.575+0000 7f248affd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:17.576 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.575+0000 7f248affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:17.576 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.575+0000 7f248affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:17.577 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.575+0000 7f248affd640 10 monclient(hunting): my global_id is 4423 2024-03-25T18:29:17.577 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.575+0000 7f248affd640 1 -- 172.21.15.155:0/2666893628 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24941398b0 con 0x7f24940a98f0 2024-03-25T18:29:17.577 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.576+0000 7f248affd640 1 -- 172.21.15.155:0/2666893628 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 350670195 0 0) 0x7f249406f730 con 0x7f24941430c0 2024-03-25T18:29:17.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.576+0000 7f248affd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:17.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.576+0000 7f248affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:17.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.576+0000 7f248affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:17.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.576+0000 7f248affd640 10 monclient(hunting): my global_id is 4433 2024-03-25T18:29:17.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.576+0000 7f248affd640 1 -- 172.21.15.155:0/2666893628 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24940711a0 con 0x7f24941430c0 2024-03-25T18:29:17.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.592+0000 7f248affd640 1 -- 172.21.15.155:0/2666893628 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1395746613 0 0) 0x7f2478004eb0 con 0x7f249413f710 2024-03-25T18:29:17.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.592+0000 7f248affd640 1 -- 172.21.15.155:0/2666893628 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f249406f730 con 0x7f249413f710 2024-03-25T18:29:17.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.592+0000 7f248affd640 1 -- 172.21.15.155:0/2666893628 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f2484001c00 con 0x7f249413f710 2024-03-25T18:29:17.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.592+0000 7f248affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:17.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.592+0000 7f248affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:29:17.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.592+0000 7f248affd640 10 monclient(hunting): dump: 2024-03-25T18:29:17.593 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:29:17.593 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:29:17.593 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:17.594 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:17.594 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:17.594 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:29:17.594 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:29:17.594 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:29:17.594 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:29:17.594 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:29:17.594 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.592+0000 7f248affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:17.595 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.594+0000 7f248affd640 1 -- 172.21.15.155:0/2666893628 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 281522456 0 0) 0x7f24940711a0 con 0x7f24941430c0 2024-03-25T18:29:17.595 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.594+0000 7f248affd640 1 -- 172.21.15.155:0/2666893628 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2478004eb0 con 0x7f24941430c0 2024-03-25T18:29:17.595 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.594+0000 7f248affd640 1 -- 172.21.15.155:0/2666893628 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f248c001a90 con 0x7f24941430c0 2024-03-25T18:29:17.595 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.594+0000 7f248affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:17.596 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.594+0000 7f248affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:17.596 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.594+0000 7f248affd640 10 monclient(hunting): dump: 2024-03-25T18:29:17.596 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:29:17.596 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:29:17.596 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:17.596 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:17.596 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:17.596 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:29:17.596 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:29:17.596 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:29:17.596 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:29:17.596 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:29:17.603 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.602+0000 7f248affd640 1 -- 172.21.15.155:0/2666893628 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3619851469 0 0) 0x7f2478004eb0 con 0x7f24941430c0 2024-03-25T18:29:17.604 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.602+0000 7f248affd640 1 -- 172.21.15.155:0/2666893628 >> v1:172.21.15.167:6789/0 conn(0x7f24940a98f0 legacy=0x7f249414bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:17.604 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.602+0000 7f248affd640 1 -- 172.21.15.155:0/2666893628 >> v1:172.21.15.155:6789/0 conn(0x7f249413f710 legacy=0x7f24941e32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:17.604 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.602+0000 7f248affd640 10 monclient: _finish_hunting 0 2024-03-25T18:29:17.604 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.602+0000 7f248affd640 1 monclient: found mon.c 2024-03-25T18:29:17.604 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.602+0000 7f248affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:17.604 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.602+0000 7f248affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:29:17.604 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.602+0000 7f248affd640 1 -- 172.21.15.155:0/2666893628 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f249406fa10 con 0x7f24941430c0 2024-03-25T18:29:17.604 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.602+0000 7f248affd640 10 monclient: _finish_auth 0 2024-03-25T18:29:17.604 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.602+0000 7f248affd640 10 monclient: _check_auth_tickets 2024-03-25T18:29:17.604 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.602+0000 7f248affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:17.604 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.602+0000 7f249c33b640 5 monclient: authenticate success, global_id 4433 2024-03-25T18:29:17.604 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.602+0000 7f249c33b640 10 monclient: _renew_subs 2024-03-25T18:29:17.605 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.602+0000 7f249c33b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:29:17.605 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.602+0000 7f249c33b640 1 -- 172.21.15.155:0/2666893628 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f249406fcf0 con 0x7f24941430c0 2024-03-25T18:29:17.605 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.602+0000 7f249c33b640 10 client.4433.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:17.605 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.602+0000 7f249c33b640 10 monclient: _renew_subs 2024-03-25T18:29:17.605 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.602+0000 7f249c33b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:29:17.605 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.602+0000 7f249c33b640 1 -- 172.21.15.155:0/2666893628 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f249414c3c0 con 0x7f24941430c0 2024-03-25T18:29:17.605 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.602+0000 7f249c33b640 1 librados: init done 2024-03-25T18:29:17.606 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.603+0000 7f249c33b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:17.606 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.603+0000 7f249c33b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:17.606 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.603+0000 7f249c33b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:29:17.606 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.603+0000 7f249c33b640 1 -- 172.21.15.155:0/2666893628 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24940627b0 con 0x7f24941430c0 2024-03-25T18:29:17.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.916+0000 7f248affd640 1 -- 172.21.15.155:0/2666893628 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f248c001fa0 con 0x7f24941430c0 2024-03-25T18:29:17.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.916+0000 7f248affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:17.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.916+0000 7f248affd640 1 -- 172.21.15.155:0/2666893628 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f248c003b10 con 0x7f24941430c0 2024-03-25T18:29:17.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.916+0000 7f248affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:17.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.916+0000 7f248affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:17.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.916+0000 7f248affd640 10 monclient: dump: 2024-03-25T18:29:17.918 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:29:17.918 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:29:17.918 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:17.918 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:17.918 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:17.918 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:29:17.918 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:29:17.918 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:29:17.918 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:29:17.918 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:29:17.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.916+0000 7f24988d4640 4 set_mon_vals no callback set 2024-03-25T18:29:17.919 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.917+0000 7f248affd640 1 -- 172.21.15.155:0/2666893628 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7f248c003d30 con 0x7f24941430c0 2024-03-25T18:29:17.919 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.917+0000 7f249a0d7640 10 monclient: get_auth_request con 0x7f247806dfb0 auth_method 0 2024-03-25T18:29:17.919 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.918+0000 7f248affd640 10 client.4433.objecter ms_handle_connect 0x7f247806dfb0 2024-03-25T18:29:17.923 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.922+0000 7f248affd640 1 -- 172.21.15.155:0/2666893628 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7f248c0537a0 con 0x7f24941430c0 2024-03-25T18:29:17.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.922+0000 7f248affd640 10 client.4433.objecter ms_dispatch 0x7f2494000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:17.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.922+0000 7f248affd640 3 client.4433.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:17.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.922+0000 7f248affd640 3 client.4433.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:17.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.922+0000 7f248affd640 20 client.4433.objecter dump_active .. 0 homeless 2024-03-25T18:29:17.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.922+0000 7f248affd640 1 -- 172.21.15.155:0/2666893628 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f248c04f2b0 con 0x7f24941430c0 2024-03-25T18:29:17.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.922+0000 7f248affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:17.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:17.922+0000 7f248affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:18.170 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:18.169+0000 7f249c33b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:18.170 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:18.169+0000 7f249c33b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:18.171 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:18.169+0000 7f249c33b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:29:18.171 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:18.169+0000 7f249c33b640 1 -- 172.21.15.155:0/2666893628 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2494069f40 con 0x7f24941430c0 2024-03-25T18:29:18.346 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:18.345+0000 7f248affd640 1 -- 172.21.15.155:0/2666893628 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7f248c0511e0 con 0x7f24941430c0 2024-03-25T18:29:18.346 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:18.345+0000 7f248affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:18.346 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:18.345+0000 7f248affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:18.347 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:29:18.347 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:18.349 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:18.347+0000 7f249c33b640 10 librados: watch_flush enter 2024-03-25T18:29:18.349 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:18.347+0000 7f249c33b640 10 librados: watch_flush exit 2024-03-25T18:29:18.349 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:18.347+0000 7f249c33b640 20 client.4433.objecter shutdown clearing up homeless session... 2024-03-25T18:29:18.349 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:18.347+0000 7f249c33b640 10 client.4433.objecter successfully canceled tick 2024-03-25T18:29:18.349 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:18.347+0000 7f249c33b640 1 -- 172.21.15.155:0/2666893628 >> v1:172.21.15.167:6800/2919070503 conn(0x7f247806dfb0 legacy=0x7f2478070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:18.349 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:18.347+0000 7f249c33b640 10 monclient: shutdown 2024-03-25T18:29:18.349 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:18.347+0000 7f249c33b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:18.349 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:18.347+0000 7f249c33b640 1 -- 172.21.15.155:0/2666893628 >> v1:172.21.15.155:6790/0 conn(0x7f24941430c0 legacy=0x7f24941e69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:18.350 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:18.348+0000 7f249c33b640 1 -- 172.21.15.155:0/2666893628 shutdown_connections 2024-03-25T18:29:18.350 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:18.348+0000 7f249c33b640 1 -- 172.21.15.155:0/2666893628 >> 172.21.15.155:0/2666893628 conn(0x7f24940b7ea0 msgr2=0x7f2494074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:18.350 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:18.348+0000 7f249c33b640 1 -- 172.21.15.155:0/2666893628 shutdown_connections 2024-03-25T18:29:18.350 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:18.348+0000 7f249c33b640 1 -- 172.21.15.155:0/2666893628 wait complete. 2024-03-25T18:29:18.350 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:18.348+0000 7f249c33b640 1 librados: shutdown 2024-03-25T18:29:18.363 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:21.231 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:04:54.835 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:21.237 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:04:54.841 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:21.278 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:04:54.882 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:21.287 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:04:54.866 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:21.290 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:04:54.895 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:21.291 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:04:54.869 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:21.310 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:04:54.914 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:21.315 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:04:54.893 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:21.336 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:04:54.941 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:25.170 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:25.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.278+0000 7fbf75625640 10 monclient: get_monmap_and_config 2024-03-25T18:29:25.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.278+0000 7fbf75625640 10 monclient: build_initial_monmap 2024-03-25T18:29:25.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.278+0000 7fbf75625640 10 monclient: monmap: 2024-03-25T18:29:25.280 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:29:25.280 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:29:25.280 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:29:25.279296+0000 2024-03-25T18:29:25.280 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:29:25.279296+0000 2024-03-25T18:29:25.280 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:25.280 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:29:25.280 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:29:25.280 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:29:25.280 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:29:25.280 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:29:25.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.279+0000 7fbf75625640 1 Processor -- start 2024-03-25T18:29:25.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.279+0000 7fbf75625640 1 -- start start 2024-03-25T18:29:25.282 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.279+0000 7fbf75625640 10 monclient: init 2024-03-25T18:29:25.282 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.280+0000 7fbf75625640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:25.282 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.280+0000 7fbf75625640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:29:25.282 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.280+0000 7fbf75625640 10 monclient(hunting): picked mon.noname-c con 0x7fbf700a98f0 addr v1:172.21.15.167:6789/0 2024-03-25T18:29:25.282 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.280+0000 7fbf75625640 10 monclient(hunting): picked mon.noname-a con 0x7fbf7013f710 addr v1:172.21.15.155:6789/0 2024-03-25T18:29:25.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.280+0000 7fbf75625640 10 monclient(hunting): picked mon.noname-b con 0x7fbf701430c0 addr v1:172.21.15.155:6790/0 2024-03-25T18:29:25.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.280+0000 7fbf75625640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf70142520 con 0x7fbf7013f710 2024-03-25T18:29:25.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.280+0000 7fbf75625640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf700b28a0 con 0x7fbf700a98f0 2024-03-25T18:29:25.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.280+0000 7fbf75625640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf700b2a30 con 0x7fbf701430c0 2024-03-25T18:29:25.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.280+0000 7fbf75625640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:25.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.280+0000 7fbf75625640 10 monclient(hunting): authenticate will time out at 7811.589933s 2024-03-25T18:29:25.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.282+0000 7fbf67fff640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fbf7013f710 0x7fbf701429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:34750/0 (socket says 172.21.15.155:34750) 2024-03-25T18:29:25.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.282+0000 7fbf67fff640 1 -- 172.21.15.155:0/1996388696 learned_addr learned my addr 172.21.15.155:0/1996388696 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:29:25.306 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.304+0000 7fbf6e7fc640 1 -- 172.21.15.155:0/1996388696 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2997079585 0 0) 0x7fbf70142520 con 0x7fbf7013f710 2024-03-25T18:29:25.306 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.304+0000 7fbf6e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:29:25.306 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.304+0000 7fbf6e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:25.306 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.304+0000 7fbf6e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:25.306 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.304+0000 7fbf6e7fc640 10 monclient(hunting): my global_id is 4431 2024-03-25T18:29:25.306 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.305+0000 7fbf6e7fc640 1 -- 172.21.15.155:0/1996388696 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf5c003520 con 0x7fbf7013f710 2024-03-25T18:29:25.312 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.311+0000 7fbf6e7fc640 1 -- 172.21.15.155:0/1996388696 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 864694753 0 0) 0x7fbf700b28a0 con 0x7fbf700a98f0 2024-03-25T18:29:25.312 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.311+0000 7fbf6e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:29:25.312 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.311+0000 7fbf6e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:25.312 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.311+0000 7fbf6e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:25.312 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.311+0000 7fbf6e7fc640 10 monclient(hunting): my global_id is 4426 2024-03-25T18:29:25.312 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.311+0000 7fbf6e7fc640 1 -- 172.21.15.155:0/1996388696 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf70142520 con 0x7fbf700a98f0 2024-03-25T18:29:25.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.317+0000 7fbf6e7fc640 1 -- 172.21.15.155:0/1996388696 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 802440680 0 0) 0x7fbf700b2a30 con 0x7fbf701430c0 2024-03-25T18:29:25.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.317+0000 7fbf6e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:29:25.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.317+0000 7fbf6e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:25.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.317+0000 7fbf6e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:25.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.317+0000 7fbf6e7fc640 10 monclient(hunting): my global_id is 4436 2024-03-25T18:29:25.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.317+0000 7fbf6e7fc640 1 -- 172.21.15.155:0/1996388696 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf700b28a0 con 0x7fbf701430c0 2024-03-25T18:29:25.328 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.326+0000 7fbf6e7fc640 1 -- 172.21.15.155:0/1996388696 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 43032701 0 0) 0x7fbf70142520 con 0x7fbf700a98f0 2024-03-25T18:29:25.328 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.326+0000 7fbf6e7fc640 1 -- 172.21.15.155:0/1996388696 >> v1:172.21.15.155:6790/0 conn(0x7fbf701430c0 legacy=0x7fbf7014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:25.328 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.326+0000 7fbf6e7fc640 1 -- 172.21.15.155:0/1996388696 >> v1:172.21.15.155:6789/0 conn(0x7fbf7013f710 legacy=0x7fbf701429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:25.328 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.326+0000 7fbf6e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:29:25.328 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.326+0000 7fbf6e7fc640 1 monclient: found mon.noname-c 2024-03-25T18:29:25.329 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.326+0000 7fbf6e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:25.329 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.326+0000 7fbf6e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.167:6789/0 2024-03-25T18:29:25.329 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.326+0000 7fbf6e7fc640 1 -- 172.21.15.155:0/1996388696 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf700b44c0 con 0x7fbf700a98f0 2024-03-25T18:29:25.329 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.326+0000 7fbf6e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:29:25.329 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.326+0000 7fbf6e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:29:25.329 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.326+0000 7fbf6e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:25.329 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.326+0000 7fbf6e7fc640 1 -- 172.21.15.155:0/1996388696 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fbf60001670 con 0x7fbf700a98f0 2024-03-25T18:29:25.329 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.326+0000 7fbf6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:25.329 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.326+0000 7fbf6e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:29:25.329 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.326+0000 7fbf6e7fc640 10 monclient: dump: 2024-03-25T18:29:25.329 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:29:25.329 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:29:25.329 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:25.329 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:25.330 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:25.330 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:29:25.330 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:29:25.330 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:29:25.330 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:29:25.330 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:29:25.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.326+0000 7fbf75625640 5 monclient: authenticate success, global_id 4426 2024-03-25T18:29:25.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.326+0000 7fbf75625640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:25.340 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.339+0000 7fbf6e7fc640 1 -- 172.21.15.155:0/1996388696 <== mon.1 v1:172.21.15.167:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf600017d0 con 0x7fbf700a98f0 2024-03-25T18:29:25.340 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.339+0000 7fbf6e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:25.340 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.339+0000 7fbf75625640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:25.340 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.339+0000 7fbf75625640 4 set_mon_vals no callback set 2024-03-25T18:29:25.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.339+0000 7fbf6e7fc640 1 -- 172.21.15.155:0/1996388696 <== mon.1 v1:172.21.15.167:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fbf60001be0 con 0x7fbf700a98f0 2024-03-25T18:29:25.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.339+0000 7fbf6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:25.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.339+0000 7fbf6e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:25.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.339+0000 7fbf6e7fc640 10 monclient: dump: 2024-03-25T18:29:25.341 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:29:25.341 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:29:25.342 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:25.342 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:25.342 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:25.342 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:29:25.342 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:29:25.342 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:29:25.342 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:29:25.342 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:29:25.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.340+0000 7fbf75625640 10 monclient: shutdown 2024-03-25T18:29:25.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.340+0000 7fbf75625640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:25.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.340+0000 7fbf75625640 1 -- 172.21.15.155:0/1996388696 >> v1:172.21.15.167:6789/0 conn(0x7fbf700a98f0 legacy=0x7fbf700a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:25.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.340+0000 7fbf75625640 1 -- 172.21.15.155:0/1996388696 shutdown_connections 2024-03-25T18:29:25.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.340+0000 7fbf75625640 1 -- 172.21.15.155:0/1996388696 >> 172.21.15.155:0/1996388696 conn(0x7fbf700b7ea0 msgr2=0x7fbf700b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:25.343 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.340+0000 7fbf75625640 1 -- 172.21.15.155:0/1996388696 shutdown_connections 2024-03-25T18:29:25.343 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.340+0000 7fbf75625640 1 -- 172.21.15.155:0/1996388696 wait complete. 2024-03-25T18:29:25.344 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.341+0000 7fbf75625640 10 monclient: build_initial_monmap 2024-03-25T18:29:25.344 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.341+0000 7fbf75625640 10 monclient: monmap: 2024-03-25T18:29:25.344 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:29:25.345 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:25.345 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:29:25.345 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:29:25.345 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:25.345 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:29:25.345 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:29:25.345 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:29:25.345 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:29:25.345 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:29:25.345 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.341+0000 7fbf75625640 1 librados: starting msgr at 2024-03-25T18:29:25.345 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.341+0000 7fbf75625640 1 librados: starting objecter 2024-03-25T18:29:25.345 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.341+0000 7fbf75625640 1 Processor -- start 2024-03-25T18:29:25.345 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.341+0000 7fbf75625640 1 -- start start 2024-03-25T18:29:25.345 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.341+0000 7fbf75625640 1 librados: setting wanted keys 2024-03-25T18:29:25.345 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.341+0000 7fbf75625640 1 librados: calling monclient init 2024-03-25T18:29:25.346 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.341+0000 7fbf75625640 10 monclient: init 2024-03-25T18:29:25.346 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.342+0000 7fbf75625640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:25.346 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.342+0000 7fbf75625640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:29:25.346 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.342+0000 7fbf75625640 10 monclient(hunting): picked mon.noname-a con 0x7fbf700a98f0 addr v1:172.21.15.155:6789/0 2024-03-25T18:29:25.346 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.342+0000 7fbf75625640 10 monclient(hunting): picked mon.noname-b con 0x7fbf7013f710 addr v1:172.21.15.167:6789/0 2024-03-25T18:29:25.346 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.342+0000 7fbf75625640 10 monclient(hunting): picked mon.noname-c con 0x7fbf701430c0 addr v1:172.21.15.155:6790/0 2024-03-25T18:29:25.346 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.342+0000 7fbf75625640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf70139480 con 0x7fbf700a98f0 2024-03-25T18:29:25.346 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.342+0000 7fbf75625640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf700ace20 con 0x7fbf7013f710 2024-03-25T18:29:25.346 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.342+0000 7fbf75625640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf70136460 con 0x7fbf701430c0 2024-03-25T18:29:25.346 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.342+0000 7fbf75625640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:25.346 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.342+0000 7fbf75625640 10 monclient(hunting): authenticate will time out at 7811.652337s 2024-03-25T18:29:25.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.357+0000 7fbf6f7fe640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fbf701430c0 0x7fbf70135d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:41112/0 (socket says 172.21.15.155:41112) 2024-03-25T18:29:25.359 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.357+0000 7fbf6f7fe640 1 -- 172.21.15.155:0/1145730651 learned_addr learned my addr 172.21.15.155:0/1145730651 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:29:25.362 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.362+0000 7fbf677fe640 10 client.?.objecter ms_handle_connect 0x7fbf701430c0 2024-03-25T18:29:25.363 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.362+0000 7fbf677fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:25.364 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.364+0000 7fbf677fe640 10 client.?.objecter ms_handle_connect 0x7fbf7013f710 2024-03-25T18:29:25.365 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.364+0000 7fbf677fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:25.367 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.366+0000 7fbf677fe640 10 client.?.objecter ms_handle_connect 0x7fbf700a98f0 2024-03-25T18:29:25.367 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.366+0000 7fbf677fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:25.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.369+0000 7fbf677fe640 1 -- 172.21.15.155:0/1145730651 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2968077242 0 0) 0x7fbf70136460 con 0x7fbf701430c0 2024-03-25T18:29:25.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.369+0000 7fbf677fe640 10 monclient(hunting): _negotiate 2024-03-25T18:29:25.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.369+0000 7fbf677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:25.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.369+0000 7fbf677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:25.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.369+0000 7fbf677fe640 10 monclient(hunting): my global_id is 4439 2024-03-25T18:29:25.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.369+0000 7fbf677fe640 1 -- 172.21.15.155:0/1145730651 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf50004eb0 con 0x7fbf701430c0 2024-03-25T18:29:25.372 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.371+0000 7fbf677fe640 1 -- 172.21.15.155:0/1145730651 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3419157269 0 0) 0x7fbf700ace20 con 0x7fbf7013f710 2024-03-25T18:29:25.372 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.371+0000 7fbf677fe640 10 monclient(hunting): _negotiate 2024-03-25T18:29:25.372 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.371+0000 7fbf677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:25.372 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.371+0000 7fbf677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:25.372 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.371+0000 7fbf677fe640 10 monclient(hunting): my global_id is 4429 2024-03-25T18:29:25.372 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.371+0000 7fbf677fe640 1 -- 172.21.15.155:0/1145730651 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf70136460 con 0x7fbf7013f710 2024-03-25T18:29:25.382 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.381+0000 7fbf677fe640 1 -- 172.21.15.155:0/1145730651 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3656799070 0 0) 0x7fbf50004eb0 con 0x7fbf701430c0 2024-03-25T18:29:25.382 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.381+0000 7fbf677fe640 1 -- 172.21.15.155:0/1145730651 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf700ace20 con 0x7fbf701430c0 2024-03-25T18:29:25.383 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.381+0000 7fbf677fe640 1 -- 172.21.15.155:0/1145730651 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fbf68001c70 con 0x7fbf701430c0 2024-03-25T18:29:25.383 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.381+0000 7fbf677fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:25.383 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.381+0000 7fbf677fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:29:25.383 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.381+0000 7fbf677fe640 10 monclient(hunting): dump: 2024-03-25T18:29:25.383 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:29:25.383 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:29:25.383 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:25.383 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:25.383 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:25.383 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:29:25.383 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:29:25.383 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:29:25.384 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:29:25.384 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:29:25.384 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.381+0000 7fbf677fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:25.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.388+0000 7fbf677fe640 1 -- 172.21.15.155:0/1145730651 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2796051468 0 0) 0x7fbf70139480 con 0x7fbf700a98f0 2024-03-25T18:29:25.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.388+0000 7fbf677fe640 10 monclient(hunting): _negotiate 2024-03-25T18:29:25.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.388+0000 7fbf677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:25.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.388+0000 7fbf677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:25.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.388+0000 7fbf677fe640 10 monclient(hunting): my global_id is 4434 2024-03-25T18:29:25.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.388+0000 7fbf677fe640 1 -- 172.21.15.155:0/1145730651 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf50004eb0 con 0x7fbf700a98f0 2024-03-25T18:29:25.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.397+0000 7fbf677fe640 1 -- 172.21.15.155:0/1145730651 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2951400190 0 0) 0x7fbf700ace20 con 0x7fbf701430c0 2024-03-25T18:29:25.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.397+0000 7fbf677fe640 1 -- 172.21.15.155:0/1145730651 >> v1:172.21.15.167:6789/0 conn(0x7fbf7013f710 legacy=0x7fbf70135640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:25.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.397+0000 7fbf677fe640 1 -- 172.21.15.155:0/1145730651 >> v1:172.21.15.155:6789/0 conn(0x7fbf700a98f0 legacy=0x7fbf700ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:25.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.397+0000 7fbf677fe640 10 monclient: _finish_hunting 0 2024-03-25T18:29:25.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.397+0000 7fbf677fe640 1 monclient: found mon.c 2024-03-25T18:29:25.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.397+0000 7fbf677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:25.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.397+0000 7fbf677fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:29:25.400 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.397+0000 7fbf677fe640 1 -- 172.21.15.155:0/1145730651 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf700ac010 con 0x7fbf701430c0 2024-03-25T18:29:25.400 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.397+0000 7fbf677fe640 10 monclient: _finish_auth 0 2024-03-25T18:29:25.400 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.397+0000 7fbf677fe640 10 monclient: _check_auth_tickets 2024-03-25T18:29:25.400 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.397+0000 7fbf677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:25.400 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.397+0000 7fbf75625640 5 monclient: authenticate success, global_id 4439 2024-03-25T18:29:25.400 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.397+0000 7fbf75625640 10 monclient: _renew_subs 2024-03-25T18:29:25.400 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.397+0000 7fbf75625640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:29:25.400 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.397+0000 7fbf75625640 1 -- 172.21.15.155:0/1145730651 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf700ac2f0 con 0x7fbf701430c0 2024-03-25T18:29:25.400 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.397+0000 7fbf75625640 10 client.4439.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:25.400 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.398+0000 7fbf75625640 10 monclient: _renew_subs 2024-03-25T18:29:25.400 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.398+0000 7fbf75625640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:29:25.400 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.398+0000 7fbf75625640 1 -- 172.21.15.155:0/1145730651 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf701381e0 con 0x7fbf701430c0 2024-03-25T18:29:25.400 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.398+0000 7fbf75625640 1 librados: init done 2024-03-25T18:29:25.401 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.398+0000 7fbf75625640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:25.401 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.398+0000 7fbf75625640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:25.401 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.398+0000 7fbf75625640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:29:25.401 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.399+0000 7fbf75625640 1 -- 172.21.15.155:0/1145730651 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf700627b0 con 0x7fbf701430c0 2024-03-25T18:29:25.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.713+0000 7fbf677fe640 1 -- 172.21.15.155:0/1145730651 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf68003990 con 0x7fbf701430c0 2024-03-25T18:29:25.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.714+0000 7fbf677fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:25.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.714+0000 7fbf677fe640 1 -- 172.21.15.155:0/1145730651 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fbf68003da0 con 0x7fbf701430c0 2024-03-25T18:29:25.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.714+0000 7fbf677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:25.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.714+0000 7fbf677fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:25.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.714+0000 7fbf677fe640 10 monclient: dump: 2024-03-25T18:29:25.715 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:29:25.715 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:29:25.716 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:25.716 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:25.716 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:25.716 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:29:25.716 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:29:25.716 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:29:25.716 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:29:25.716 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:29:25.716 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.714+0000 7fbf6dffb640 4 set_mon_vals no callback set 2024-03-25T18:29:25.717 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.714+0000 7fbf677fe640 1 -- 172.21.15.155:0/1145730651 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7fbf68003fc0 con 0x7fbf701430c0 2024-03-25T18:29:25.717 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.715+0000 7fbf6effd640 10 monclient: get_auth_request con 0x7fbf5006de00 auth_method 0 2024-03-25T18:29:25.717 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.715+0000 7fbf677fe640 10 client.4439.objecter ms_handle_connect 0x7fbf5006de00 2024-03-25T18:29:25.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.723+0000 7fbf677fe640 1 -- 172.21.15.155:0/1145730651 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7fbf68054ae0 con 0x7fbf701430c0 2024-03-25T18:29:25.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.723+0000 7fbf677fe640 10 client.4439.objecter ms_dispatch 0x7fbf70000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:25.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.723+0000 7fbf677fe640 3 client.4439.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:25.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.723+0000 7fbf677fe640 3 client.4439.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:25.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.723+0000 7fbf677fe640 20 client.4439.objecter dump_active .. 0 homeless 2024-03-25T18:29:25.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.723+0000 7fbf677fe640 1 -- 172.21.15.155:0/1145730651 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbf680505f0 con 0x7fbf701430c0 2024-03-25T18:29:25.725 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.723+0000 7fbf677fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:25.725 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.723+0000 7fbf677fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:25.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.984+0000 7fbf75625640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:25.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.984+0000 7fbf75625640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:25.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.984+0000 7fbf75625640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:29:25.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:25.984+0000 7fbf75625640 1 -- 172.21.15.155:0/1145730651 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbf70141b00 con 0x7fbf701430c0 2024-03-25T18:29:26.004 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:26.003+0000 7fbf677fe640 1 -- 172.21.15.155:0/1145730651 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7fbf68052520 con 0x7fbf701430c0 2024-03-25T18:29:26.004 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:26.003+0000 7fbf677fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:26.004 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:26.003+0000 7fbf677fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:26.005 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:29:26.005 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:26.007 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:26.005+0000 7fbf75625640 10 librados: watch_flush enter 2024-03-25T18:29:26.007 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:26.006+0000 7fbf75625640 10 librados: watch_flush exit 2024-03-25T18:29:26.007 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:26.006+0000 7fbf75625640 20 client.4439.objecter shutdown clearing up homeless session... 2024-03-25T18:29:26.007 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:26.006+0000 7fbf75625640 10 client.4439.objecter successfully canceled tick 2024-03-25T18:29:26.007 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:26.006+0000 7fbf75625640 1 -- 172.21.15.155:0/1145730651 >> v1:172.21.15.167:6800/2919070503 conn(0x7fbf5006de00 legacy=0x7fbf500702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:26.007 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:26.006+0000 7fbf75625640 10 monclient: shutdown 2024-03-25T18:29:26.007 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:26.006+0000 7fbf75625640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:26.007 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:26.006+0000 7fbf75625640 1 -- 172.21.15.155:0/1145730651 >> v1:172.21.15.155:6790/0 conn(0x7fbf701430c0 legacy=0x7fbf70135d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:26.008 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:26.006+0000 7fbf75625640 1 -- 172.21.15.155:0/1145730651 shutdown_connections 2024-03-25T18:29:26.008 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:26.006+0000 7fbf75625640 1 -- 172.21.15.155:0/1145730651 >> 172.21.15.155:0/1145730651 conn(0x7fbf700b7ea0 msgr2=0x7fbf70074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:26.008 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:26.006+0000 7fbf75625640 1 -- 172.21.15.155:0/1145730651 shutdown_connections 2024-03-25T18:29:26.008 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:26.006+0000 7fbf75625640 1 -- 172.21.15.155:0/1145730651 wait complete. 2024-03-25T18:29:26.008 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:26.006+0000 7fbf75625640 1 librados: shutdown 2024-03-25T18:29:26.020 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:32.827 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:32.942 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.941+0000 7f2d44cd0640 10 monclient: get_monmap_and_config 2024-03-25T18:29:32.943 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.941+0000 7f2d44cd0640 10 monclient: build_initial_monmap 2024-03-25T18:29:32.943 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.941+0000 7f2d44cd0640 10 monclient: monmap: 2024-03-25T18:29:32.943 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:29:32.943 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:29:32.943 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:29:32.942777+0000 2024-03-25T18:29:32.943 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:29:32.942777+0000 2024-03-25T18:29:32.943 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:32.943 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:29:32.943 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:29:32.943 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:29:32.943 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:29:32.944 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:29:32.945 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.942+0000 7f2d44cd0640 1 Processor -- start 2024-03-25T18:29:32.945 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.942+0000 7f2d44cd0640 1 -- start start 2024-03-25T18:29:32.945 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.942+0000 7f2d44cd0640 10 monclient: init 2024-03-25T18:29:32.945 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.943+0000 7f2d44cd0640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:32.945 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.943+0000 7f2d44cd0640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:29:32.945 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.943+0000 7f2d44cd0640 10 monclient(hunting): picked mon.noname-b con 0x7f2d400a98f0 addr v1:172.21.15.155:6790/0 2024-03-25T18:29:32.945 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.943+0000 7f2d44cd0640 10 monclient(hunting): picked mon.noname-a con 0x7f2d4013f710 addr v1:172.21.15.155:6789/0 2024-03-25T18:29:32.945 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.943+0000 7f2d44cd0640 10 monclient(hunting): picked mon.noname-c con 0x7f2d401430c0 addr v1:172.21.15.167:6789/0 2024-03-25T18:29:32.945 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.943+0000 7f2d44cd0640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d40142520 con 0x7f2d4013f710 2024-03-25T18:29:32.945 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.943+0000 7f2d44cd0640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d400b28a0 con 0x7f2d401430c0 2024-03-25T18:29:32.945 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.943+0000 7f2d44cd0640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d400b2a30 con 0x7f2d400a98f0 2024-03-25T18:29:32.945 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.943+0000 7f2d44cd0640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:32.945 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.943+0000 7f2d44cd0640 10 monclient(hunting): authenticate will time out at 7819.253707s 2024-03-25T18:29:32.951 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.950+0000 7f2d3ed9d640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f2d401430c0 0x7f2d4014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:38716/0 (socket says 172.21.15.155:38716) 2024-03-25T18:29:32.951 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.950+0000 7f2d3ed9d640 1 -- 172.21.15.155:0/3552739348 learned_addr learned my addr 172.21.15.155:0/3552739348 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:29:32.958 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.957+0000 7f2d3dd9b640 1 -- 172.21.15.155:0/3552739348 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2306349286 0 0) 0x7f2d40142520 con 0x7f2d4013f710 2024-03-25T18:29:32.958 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.957+0000 7f2d3dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:29:32.958 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.957+0000 7f2d3dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:32.958 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.957+0000 7f2d3dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:32.958 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.957+0000 7f2d3dd9b640 10 monclient(hunting): my global_id is 4437 2024-03-25T18:29:32.958 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.957+0000 7f2d3dd9b640 1 -- 172.21.15.155:0/3552739348 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d2c003520 con 0x7f2d4013f710 2024-03-25T18:29:32.966 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.965+0000 7f2d3dd9b640 1 -- 172.21.15.155:0/3552739348 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2744793640 0 0) 0x7f2d400b2a30 con 0x7f2d400a98f0 2024-03-25T18:29:32.966 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.965+0000 7f2d3dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:29:32.967 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.965+0000 7f2d3dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:32.967 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.965+0000 7f2d3dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:32.967 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.965+0000 7f2d3dd9b640 10 monclient(hunting): my global_id is 4442 2024-03-25T18:29:32.967 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.965+0000 7f2d3dd9b640 1 -- 172.21.15.155:0/3552739348 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d40142520 con 0x7f2d400a98f0 2024-03-25T18:29:32.969 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.968+0000 7f2d3dd9b640 1 -- 172.21.15.155:0/3552739348 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2711630116 0 0) 0x7f2d400b28a0 con 0x7f2d401430c0 2024-03-25T18:29:32.969 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.968+0000 7f2d3dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:29:32.969 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.968+0000 7f2d3dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:32.969 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.968+0000 7f2d3dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:32.969 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.968+0000 7f2d3dd9b640 10 monclient(hunting): my global_id is 4432 2024-03-25T18:29:32.969 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.968+0000 7f2d3dd9b640 1 -- 172.21.15.155:0/3552739348 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d400b2a30 con 0x7f2d401430c0 2024-03-25T18:29:32.975 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.973+0000 7f2d3dd9b640 1 -- 172.21.15.155:0/3552739348 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2769669539 0 0) 0x7f2d2c003520 con 0x7f2d4013f710 2024-03-25T18:29:32.975 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.973+0000 7f2d3dd9b640 1 -- 172.21.15.155:0/3552739348 >> v1:172.21.15.155:6790/0 conn(0x7f2d400a98f0 legacy=0x7f2d400a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:32.975 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.973+0000 7f2d3dd9b640 1 -- 172.21.15.155:0/3552739348 >> v1:172.21.15.167:6789/0 conn(0x7f2d401430c0 legacy=0x7f2d4014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:32.975 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.973+0000 7f2d3dd9b640 10 monclient: _finish_hunting 0 2024-03-25T18:29:32.975 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.973+0000 7f2d3dd9b640 1 monclient: found mon.noname-a 2024-03-25T18:29:32.975 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.973+0000 7f2d3dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:32.975 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.973+0000 7f2d3dd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-25T18:29:32.975 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.973+0000 7f2d3dd9b640 1 -- 172.21.15.155:0/3552739348 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d400b44c0 con 0x7f2d4013f710 2024-03-25T18:29:32.975 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.973+0000 7f2d3dd9b640 10 monclient: _finish_auth 0 2024-03-25T18:29:32.975 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.973+0000 7f2d3dd9b640 10 monclient: _check_auth_tickets 2024-03-25T18:29:32.975 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.973+0000 7f2d3dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:32.975 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.973+0000 7f2d3dd9b640 1 -- 172.21.15.155:0/3552739348 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f2d28001690 con 0x7f2d4013f710 2024-03-25T18:29:32.975 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.973+0000 7f2d44cd0640 5 monclient: authenticate success, global_id 4437 2024-03-25T18:29:32.976 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.973+0000 7f2d44cd0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:32.976 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.973+0000 7f2d3dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:32.976 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.973+0000 7f2d3dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:29:32.976 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.973+0000 7f2d3dd9b640 10 monclient: dump: 2024-03-25T18:29:32.976 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:29:32.976 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:29:32.976 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:32.976 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:32.976 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:32.976 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:29:32.976 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:29:32.976 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:29:32.976 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:29:32.976 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:29:32.976 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.973+0000 7f2d44cd0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:32.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.981+0000 7f2d3dd9b640 1 -- 172.21.15.155:0/3552739348 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2d280017f0 con 0x7f2d4013f710 2024-03-25T18:29:32.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.981+0000 7f2d3dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:32.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.981+0000 7f2d3dd9b640 1 -- 172.21.15.155:0/3552739348 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f2d28001c00 con 0x7f2d4013f710 2024-03-25T18:29:32.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.981+0000 7f2d3dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:32.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.981+0000 7f2d3dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:32.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.981+0000 7f2d3dd9b640 10 monclient: dump: 2024-03-25T18:29:32.982 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:29:32.983 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:29:32.983 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:32.983 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:32.983 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:32.983 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:29:32.983 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:29:32.983 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:29:32.983 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:29:32.983 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:29:32.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.981+0000 7f2d44cd0640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:32.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.981+0000 7f2d44cd0640 4 set_mon_vals no callback set 2024-03-25T18:29:32.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.982+0000 7f2d44cd0640 10 monclient: shutdown 2024-03-25T18:29:32.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.982+0000 7f2d44cd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:32.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.982+0000 7f2d44cd0640 1 -- 172.21.15.155:0/3552739348 >> v1:172.21.15.155:6789/0 conn(0x7f2d4013f710 legacy=0x7f2d401429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:32.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.982+0000 7f2d44cd0640 1 -- 172.21.15.155:0/3552739348 shutdown_connections 2024-03-25T18:29:32.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.982+0000 7f2d44cd0640 1 -- 172.21.15.155:0/3552739348 >> 172.21.15.155:0/3552739348 conn(0x7f2d400b7ea0 msgr2=0x7f2d400b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:32.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.982+0000 7f2d44cd0640 1 -- 172.21.15.155:0/3552739348 shutdown_connections 2024-03-25T18:29:32.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.982+0000 7f2d44cd0640 1 -- 172.21.15.155:0/3552739348 wait complete. 2024-03-25T18:29:32.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.983+0000 7f2d44cd0640 10 monclient: build_initial_monmap 2024-03-25T18:29:32.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.983+0000 7f2d44cd0640 10 monclient: monmap: 2024-03-25T18:29:32.986 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:29:32.986 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:32.986 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:29:32.986 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:29:32.986 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:32.986 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:29:32.986 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:29:32.987 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:29:32.987 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:29:32.987 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:29:32.987 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.983+0000 7f2d44cd0640 1 librados: starting msgr at 2024-03-25T18:29:32.987 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.983+0000 7f2d44cd0640 1 librados: starting objecter 2024-03-25T18:29:32.987 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.983+0000 7f2d44cd0640 1 Processor -- start 2024-03-25T18:29:32.987 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.984+0000 7f2d44cd0640 1 -- start start 2024-03-25T18:29:32.987 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.984+0000 7f2d44cd0640 1 librados: setting wanted keys 2024-03-25T18:29:32.987 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.984+0000 7f2d44cd0640 1 librados: calling monclient init 2024-03-25T18:29:32.987 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.984+0000 7f2d44cd0640 10 monclient: init 2024-03-25T18:29:32.987 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.984+0000 7f2d44cd0640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:32.987 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.984+0000 7f2d44cd0640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:29:32.987 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.984+0000 7f2d44cd0640 10 monclient(hunting): picked mon.noname-a con 0x7f2d400a98f0 addr v1:172.21.15.155:6789/0 2024-03-25T18:29:32.987 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.984+0000 7f2d44cd0640 10 monclient(hunting): picked mon.noname-c con 0x7f2d4013f710 addr v1:172.21.15.155:6790/0 2024-03-25T18:29:32.987 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.984+0000 7f2d44cd0640 10 monclient(hunting): picked mon.noname-b con 0x7f2d401430c0 addr v1:172.21.15.167:6789/0 2024-03-25T18:29:32.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.984+0000 7f2d44cd0640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d4006fbe0 con 0x7f2d400a98f0 2024-03-25T18:29:32.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.984+0000 7f2d44cd0640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d4014e630 con 0x7f2d401430c0 2024-03-25T18:29:32.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.984+0000 7f2d44cd0640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d4006fe50 con 0x7f2d4013f710 2024-03-25T18:29:32.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.984+0000 7f2d44cd0640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:32.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.984+0000 7f2d44cd0640 10 monclient(hunting): authenticate will time out at 7819.294679s 2024-03-25T18:29:32.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.991+0000 7f2d37fff640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f2d4013f710 0x7f2d4006dea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:43414/0 (socket says 172.21.15.155:43414) 2024-03-25T18:29:32.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.991+0000 7f2d37fff640 1 -- 172.21.15.155:0/3025490358 learned_addr learned my addr 172.21.15.155:0/3025490358 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:29:32.995 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.994+0000 7f2d36ffd640 10 client.?.objecter ms_handle_connect 0x7f2d4013f710 2024-03-25T18:29:32.995 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:32.994+0000 7f2d36ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:33.002 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.001+0000 7f2d3ed9d640 1 -- 172.21.15.155:0/3025490358 >> v1:172.21.15.167:6789/0 conn(0x7f2d401430c0 legacy=0x7f2d4006e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-25T18:29:33.002 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.001+0000 7f2d3ed9d640 1 -- 172.21.15.155:0/3025490358 >> v1:172.21.15.167:6789/0 conn(0x7f2d401430c0 legacy=0x7f2d4006e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:29:33.002 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.001+0000 7f2d3ed9d640 1 --1- 172.21.15.155:0/3025490358 >> v1:172.21.15.167:6789/0 conn(0x7f2d401430c0 0x7f2d4006e5b0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_1 read connect reply failed 2024-03-25T18:29:33.002 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.001+0000 7f2d3ed9d640 1 --1- 172.21.15.155:0/3025490358 >> v1:172.21.15.167:6789/0 conn(0x7f2d401430c0 0x7f2d4006e5b0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-03-25T18:29:33.002 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.001+0000 7f2d36ffd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.167:6789/0 2024-03-25T18:29:33.003 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.001+0000 7f2d36ffd640 1 -- 172.21.15.155:0/3025490358 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3778043105 0 0) 0x7f2d4006fe50 con 0x7f2d4013f710 2024-03-25T18:29:33.003 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.001+0000 7f2d36ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:33.003 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.001+0000 7f2d36ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:33.003 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.001+0000 7f2d36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:33.003 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.001+0000 7f2d36ffd640 10 monclient(hunting): my global_id is 4445 2024-03-25T18:29:33.003 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.001+0000 7f2d36ffd640 1 -- 172.21.15.155:0/3025490358 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d20004eb0 con 0x7f2d4013f710 2024-03-25T18:29:33.010 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.009+0000 7f2d36ffd640 10 client.?.objecter ms_handle_connect 0x7f2d400a98f0 2024-03-25T18:29:33.010 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.009+0000 7f2d36ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:33.023 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.022+0000 7f2d36ffd640 1 -- 172.21.15.155:0/3025490358 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1126153765 0 0) 0x7f2d20004eb0 con 0x7f2d4013f710 2024-03-25T18:29:33.023 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.022+0000 7f2d36ffd640 1 -- 172.21.15.155:0/3025490358 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2d4006fe50 con 0x7f2d4013f710 2024-03-25T18:29:33.023 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.022+0000 7f2d36ffd640 1 -- 172.21.15.155:0/3025490358 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f2d28001bc0 con 0x7f2d4013f710 2024-03-25T18:29:33.023 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.022+0000 7f2d36ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:33.023 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.022+0000 7f2d36ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:29:33.023 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.022+0000 7f2d36ffd640 10 monclient(hunting): dump: 2024-03-25T18:29:33.023 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:29:33.024 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:29:33.024 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:33.024 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:33.024 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:33.024 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:29:33.024 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:29:33.024 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:29:33.024 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:29:33.024 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:29:33.024 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.022+0000 7f2d36ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:33.035 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.034+0000 7f2d36ffd640 1 -- 172.21.15.155:0/3025490358 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2131823862 0 0) 0x7f2d4006fbe0 con 0x7f2d400a98f0 2024-03-25T18:29:33.035 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.034+0000 7f2d36ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:33.035 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.034+0000 7f2d36ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:33.035 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.034+0000 7f2d36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:33.035 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.034+0000 7f2d36ffd640 10 monclient(hunting): my global_id is 4440 2024-03-25T18:29:33.036 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.034+0000 7f2d36ffd640 1 -- 172.21.15.155:0/3025490358 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d20004eb0 con 0x7f2d400a98f0 2024-03-25T18:29:33.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.048+0000 7f2d36ffd640 1 -- 172.21.15.155:0/3025490358 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2719154599 0 0) 0x7f2d4006fe50 con 0x7f2d4013f710 2024-03-25T18:29:33.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.048+0000 7f2d36ffd640 1 -- 172.21.15.155:0/3025490358 >> v1:172.21.15.167:6789/0 conn(0x7f2d401430c0 legacy=0x7f2d4006e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:29:33.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.048+0000 7f2d36ffd640 1 -- 172.21.15.155:0/3025490358 >> v1:172.21.15.155:6789/0 conn(0x7f2d400a98f0 legacy=0x7f2d400708c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:33.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.048+0000 7f2d36ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:29:33.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.048+0000 7f2d36ffd640 1 monclient: found mon.c 2024-03-25T18:29:33.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.048+0000 7f2d36ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:33.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.048+0000 7f2d36ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:29:33.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.048+0000 7f2d36ffd640 1 -- 172.21.15.155:0/3025490358 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d40070130 con 0x7f2d4013f710 2024-03-25T18:29:33.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.048+0000 7f2d36ffd640 10 monclient: _finish_auth 0 2024-03-25T18:29:33.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.048+0000 7f2d36ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:29:33.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.048+0000 7f2d36ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:33.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.048+0000 7f2d44cd0640 5 monclient: authenticate success, global_id 4445 2024-03-25T18:29:33.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.048+0000 7f2d44cd0640 10 monclient: _renew_subs 2024-03-25T18:29:33.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.048+0000 7f2d44cd0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:29:33.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.048+0000 7f2d44cd0640 1 -- 172.21.15.155:0/3025490358 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d40070fd0 con 0x7f2d4013f710 2024-03-25T18:29:33.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.048+0000 7f2d44cd0640 10 client.4445.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:33.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.048+0000 7f2d44cd0640 10 monclient: _renew_subs 2024-03-25T18:29:33.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.048+0000 7f2d44cd0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:29:33.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.048+0000 7f2d44cd0640 1 -- 172.21.15.155:0/3025490358 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d4006ecc0 con 0x7f2d4013f710 2024-03-25T18:29:33.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.048+0000 7f2d44cd0640 1 librados: init done 2024-03-25T18:29:33.052 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.049+0000 7f2d44cd0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:33.052 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.049+0000 7f2d44cd0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:33.052 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.049+0000 7f2d44cd0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:29:33.052 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.049+0000 7f2d44cd0640 1 -- 172.21.15.155:0/3025490358 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d400627b0 con 0x7f2d4013f710 2024-03-25T18:29:33.372 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.370+0000 7f2d36ffd640 1 -- 172.21.15.155:0/3025490358 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2d280011f0 con 0x7f2d4013f710 2024-03-25T18:29:33.372 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.370+0000 7f2d36ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:33.372 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.370+0000 7f2d36ffd640 1 -- 172.21.15.155:0/3025490358 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f2d28003b00 con 0x7f2d4013f710 2024-03-25T18:29:33.372 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.370+0000 7f2d36ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:33.372 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.370+0000 7f2d36ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:33.372 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.370+0000 7f2d36ffd640 10 monclient: dump: 2024-03-25T18:29:33.372 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:29:33.372 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:29:33.372 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:33.372 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:33.372 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:33.372 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:29:33.373 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:29:33.373 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:29:33.373 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:29:33.373 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:29:33.373 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.370+0000 7f2d3d59a640 4 set_mon_vals no callback set 2024-03-25T18:29:33.373 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.371+0000 7f2d36ffd640 1 -- 172.21.15.155:0/3025490358 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7f2d28003d20 con 0x7f2d4013f710 2024-03-25T18:29:33.374 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.372+0000 7f2d3e59c640 10 monclient: get_auth_request con 0x7f2d2006dd30 auth_method 0 2024-03-25T18:29:33.374 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.372+0000 7f2d36ffd640 10 client.4445.objecter ms_handle_connect 0x7f2d2006dd30 2024-03-25T18:29:33.378 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.377+0000 7f2d36ffd640 1 -- 172.21.15.155:0/3025490358 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7f2d280548c0 con 0x7f2d4013f710 2024-03-25T18:29:33.378 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.377+0000 7f2d36ffd640 10 client.4445.objecter ms_dispatch 0x7f2d40000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:33.378 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.377+0000 7f2d36ffd640 3 client.4445.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:33.378 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.377+0000 7f2d36ffd640 3 client.4445.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:33.378 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.377+0000 7f2d36ffd640 20 client.4445.objecter dump_active .. 0 homeless 2024-03-25T18:29:33.378 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.377+0000 7f2d36ffd640 1 -- 172.21.15.155:0/3025490358 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2d28050200 con 0x7f2d4013f710 2024-03-25T18:29:33.379 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.377+0000 7f2d36ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:33.379 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.377+0000 7f2d36ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:33.610 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.609+0000 7f2d44cd0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:33.610 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.609+0000 7f2d44cd0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:33.610 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.609+0000 7f2d44cd0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:29:33.610 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.609+0000 7f2d44cd0640 1 -- 172.21.15.155:0/3025490358 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2d40139710 con 0x7f2d4013f710 2024-03-25T18:29:33.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.645+0000 7f2d36ffd640 1 -- 172.21.15.155:0/3025490358 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7f2d28052300 con 0x7f2d4013f710 2024-03-25T18:29:33.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.645+0000 7f2d36ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:33.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.645+0000 7f2d36ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:33.647 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:29:33.647 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:33.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.647+0000 7f2d44cd0640 10 librados: watch_flush enter 2024-03-25T18:29:33.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.647+0000 7f2d44cd0640 10 librados: watch_flush exit 2024-03-25T18:29:33.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.647+0000 7f2d44cd0640 20 client.4445.objecter shutdown clearing up homeless session... 2024-03-25T18:29:33.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.647+0000 7f2d44cd0640 10 client.4445.objecter successfully canceled tick 2024-03-25T18:29:33.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.647+0000 7f2d44cd0640 1 -- 172.21.15.155:0/3025490358 >> v1:172.21.15.167:6800/2919070503 conn(0x7f2d2006dd30 legacy=0x7f2d200701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:33.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.647+0000 7f2d44cd0640 10 monclient: shutdown 2024-03-25T18:29:33.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.647+0000 7f2d44cd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:33.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.647+0000 7f2d44cd0640 1 -- 172.21.15.155:0/3025490358 >> v1:172.21.15.155:6790/0 conn(0x7f2d4013f710 legacy=0x7f2d4006dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:33.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.647+0000 7f2d44cd0640 1 -- 172.21.15.155:0/3025490358 shutdown_connections 2024-03-25T18:29:33.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.647+0000 7f2d44cd0640 1 -- 172.21.15.155:0/3025490358 >> 172.21.15.155:0/3025490358 conn(0x7f2d400b7ea0 msgr2=0x7f2d40074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:33.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.647+0000 7f2d44cd0640 1 -- 172.21.15.155:0/3025490358 shutdown_connections 2024-03-25T18:29:33.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.647+0000 7f2d44cd0640 1 -- 172.21.15.155:0/3025490358 wait complete. 2024-03-25T18:29:33.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:33.648+0000 7f2d44cd0640 1 librados: shutdown 2024-03-25T18:29:33.662 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:40.469 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:40.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.580+0000 7fd5d1ec5640 10 monclient: get_monmap_and_config 2024-03-25T18:29:40.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.580+0000 7fd5d1ec5640 10 monclient: build_initial_monmap 2024-03-25T18:29:40.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.580+0000 7fd5d1ec5640 10 monclient: monmap: 2024-03-25T18:29:40.582 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:29:40.582 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:29:40.582 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:29:40.581831+0000 2024-03-25T18:29:40.582 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:29:40.581831+0000 2024-03-25T18:29:40.582 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:40.582 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:29:40.583 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:29:40.583 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:29:40.583 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:29:40.583 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:29:40.584 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.581+0000 7fd5d1ec5640 1 Processor -- start 2024-03-25T18:29:40.585 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.581+0000 7fd5d1ec5640 1 -- start start 2024-03-25T18:29:40.585 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.581+0000 7fd5d1ec5640 10 monclient: init 2024-03-25T18:29:40.585 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.582+0000 7fd5d1ec5640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:40.585 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.582+0000 7fd5d1ec5640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:29:40.585 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.582+0000 7fd5d1ec5640 10 monclient(hunting): picked mon.noname-a con 0x7fd5cc0a98f0 addr v1:172.21.15.155:6789/0 2024-03-25T18:29:40.585 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.582+0000 7fd5d1ec5640 10 monclient(hunting): picked mon.noname-b con 0x7fd5cc13f710 addr v1:172.21.15.155:6790/0 2024-03-25T18:29:40.585 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.582+0000 7fd5d1ec5640 10 monclient(hunting): picked mon.noname-c con 0x7fd5cc1430c0 addr v1:172.21.15.167:6789/0 2024-03-25T18:29:40.585 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.582+0000 7fd5d1ec5640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5cc142520 con 0x7fd5cc0a98f0 2024-03-25T18:29:40.585 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.582+0000 7fd5d1ec5640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5cc0b28a0 con 0x7fd5cc1430c0 2024-03-25T18:29:40.585 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.582+0000 7fd5d1ec5640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5cc0b2a30 con 0x7fd5cc13f710 2024-03-25T18:29:40.585 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.582+0000 7fd5d1ec5640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:40.585 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.582+0000 7fd5d1ec5640 10 monclient(hunting): authenticate will time out at 7826.892515s 2024-03-25T18:29:40.591 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.590+0000 7fd5cbfff640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7fd5cc1430c0 0x7fd5cc14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:46696/0 (socket says 172.21.15.155:46696) 2024-03-25T18:29:40.591 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.590+0000 7fd5cbfff640 1 -- 172.21.15.155:0/2845968174 learned_addr learned my addr 172.21.15.155:0/2845968174 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:29:40.603 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.602+0000 7fd5ca7fc640 1 -- 172.21.15.155:0/2845968174 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1061406939 0 0) 0x7fd5cc0b28a0 con 0x7fd5cc1430c0 2024-03-25T18:29:40.603 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.602+0000 7fd5ca7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:29:40.603 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.602+0000 7fd5ca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:40.603 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.602+0000 7fd5ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:40.603 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.602+0000 7fd5ca7fc640 10 monclient(hunting): my global_id is 4435 2024-03-25T18:29:40.604 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.602+0000 7fd5ca7fc640 1 -- 172.21.15.155:0/2845968174 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5b4003520 con 0x7fd5cc1430c0 2024-03-25T18:29:40.613 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.611+0000 7fd5ca7fc640 1 -- 172.21.15.155:0/2845968174 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3732173640 0 0) 0x7fd5cc0b2a30 con 0x7fd5cc13f710 2024-03-25T18:29:40.613 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.611+0000 7fd5ca7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:29:40.613 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.611+0000 7fd5ca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:40.613 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.611+0000 7fd5ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:40.613 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.611+0000 7fd5ca7fc640 10 monclient(hunting): my global_id is 4448 2024-03-25T18:29:40.613 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.611+0000 7fd5ca7fc640 1 -- 172.21.15.155:0/2845968174 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5cc0b28a0 con 0x7fd5cc13f710 2024-03-25T18:29:40.617 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.616+0000 7fd5ca7fc640 1 -- 172.21.15.155:0/2845968174 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2679427821 0 0) 0x7fd5cc0b28a0 con 0x7fd5cc13f710 2024-03-25T18:29:40.617 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.616+0000 7fd5ca7fc640 1 -- 172.21.15.155:0/2845968174 >> v1:172.21.15.167:6789/0 conn(0x7fd5cc1430c0 legacy=0x7fd5cc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:40.617 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.616+0000 7fd5ca7fc640 1 -- 172.21.15.155:0/2845968174 >> v1:172.21.15.155:6789/0 conn(0x7fd5cc0a98f0 legacy=0x7fd5cc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:40.617 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.616+0000 7fd5ca7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:29:40.617 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.616+0000 7fd5ca7fc640 1 monclient: found mon.noname-b 2024-03-25T18:29:40.618 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.616+0000 7fd5ca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:40.618 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.616+0000 7fd5ca7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-25T18:29:40.618 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.616+0000 7fd5ca7fc640 1 -- 172.21.15.155:0/2845968174 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5cc0b44c0 con 0x7fd5cc13f710 2024-03-25T18:29:40.618 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.616+0000 7fd5ca7fc640 10 monclient: _finish_auth 0 2024-03-25T18:29:40.618 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.616+0000 7fd5ca7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:29:40.618 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.616+0000 7fd5ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:40.618 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.616+0000 7fd5ca7fc640 1 -- 172.21.15.155:0/2845968174 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fd5c0001670 con 0x7fd5cc13f710 2024-03-25T18:29:40.618 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.616+0000 7fd5ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:40.618 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.616+0000 7fd5ca7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:29:40.618 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.616+0000 7fd5ca7fc640 10 monclient: dump: 2024-03-25T18:29:40.618 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:29:40.618 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:29:40.618 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:40.619 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:40.619 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:40.619 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:29:40.619 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:29:40.619 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:29:40.619 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:29:40.619 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:29:40.619 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.616+0000 7fd5d1ec5640 5 monclient: authenticate success, global_id 4448 2024-03-25T18:29:40.619 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.616+0000 7fd5d1ec5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:40.628 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.627+0000 7fd5ca7fc640 1 -- 172.21.15.155:0/2845968174 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd5c00017d0 con 0x7fd5cc13f710 2024-03-25T18:29:40.628 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.627+0000 7fd5ca7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:40.628 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.627+0000 7fd5ca7fc640 1 -- 172.21.15.155:0/2845968174 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fd5c0001ba0 con 0x7fd5cc13f710 2024-03-25T18:29:40.628 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.627+0000 7fd5ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:40.628 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.627+0000 7fd5ca7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:40.628 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.627+0000 7fd5ca7fc640 10 monclient: dump: 2024-03-25T18:29:40.628 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:29:40.628 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:29:40.628 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:40.628 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:40.629 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:40.629 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:29:40.629 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:29:40.629 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:29:40.629 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:29:40.629 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:29:40.629 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.627+0000 7fd5d1ec5640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:40.629 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.627+0000 7fd5d1ec5640 4 set_mon_vals no callback set 2024-03-25T18:29:40.631 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.628+0000 7fd5d1ec5640 10 monclient: shutdown 2024-03-25T18:29:40.631 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.628+0000 7fd5d1ec5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:40.631 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.628+0000 7fd5d1ec5640 1 -- 172.21.15.155:0/2845968174 >> v1:172.21.15.155:6790/0 conn(0x7fd5cc13f710 legacy=0x7fd5cc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:40.631 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.628+0000 7fd5d1ec5640 1 -- 172.21.15.155:0/2845968174 shutdown_connections 2024-03-25T18:29:40.631 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.628+0000 7fd5d1ec5640 1 -- 172.21.15.155:0/2845968174 >> 172.21.15.155:0/2845968174 conn(0x7fd5cc0b7ea0 msgr2=0x7fd5cc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:40.631 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.628+0000 7fd5d1ec5640 1 -- 172.21.15.155:0/2845968174 shutdown_connections 2024-03-25T18:29:40.631 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.628+0000 7fd5d1ec5640 1 -- 172.21.15.155:0/2845968174 wait complete. 2024-03-25T18:29:40.631 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.629+0000 7fd5d1ec5640 10 monclient: build_initial_monmap 2024-03-25T18:29:40.631 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.629+0000 7fd5d1ec5640 10 monclient: monmap: 2024-03-25T18:29:40.631 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:29:40.631 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:40.631 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:29:40.632 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:29:40.632 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:40.632 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:29:40.632 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:29:40.632 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:29:40.632 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:29:40.632 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:29:40.632 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.629+0000 7fd5d1ec5640 1 librados: starting msgr at 2024-03-25T18:29:40.632 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.629+0000 7fd5d1ec5640 1 librados: starting objecter 2024-03-25T18:29:40.632 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.629+0000 7fd5d1ec5640 1 Processor -- start 2024-03-25T18:29:40.632 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.630+0000 7fd5d1ec5640 1 -- start start 2024-03-25T18:29:40.632 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.630+0000 7fd5d1ec5640 1 librados: setting wanted keys 2024-03-25T18:29:40.632 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.630+0000 7fd5d1ec5640 1 librados: calling monclient init 2024-03-25T18:29:40.632 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.630+0000 7fd5d1ec5640 10 monclient: init 2024-03-25T18:29:40.633 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.630+0000 7fd5d1ec5640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:40.634 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.630+0000 7fd5d1ec5640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:29:40.634 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.630+0000 7fd5d1ec5640 10 monclient(hunting): picked mon.noname-a con 0x7fd5cc0a98f0 addr v1:172.21.15.155:6789/0 2024-03-25T18:29:40.634 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.630+0000 7fd5d1ec5640 10 monclient(hunting): picked mon.noname-b con 0x7fd5cc13f710 addr v1:172.21.15.167:6789/0 2024-03-25T18:29:40.634 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.630+0000 7fd5d1ec5640 10 monclient(hunting): picked mon.noname-c con 0x7fd5cc1430c0 addr v1:172.21.15.155:6790/0 2024-03-25T18:29:40.634 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.630+0000 7fd5d1ec5640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5cc06fc80 con 0x7fd5cc0a98f0 2024-03-25T18:29:40.634 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.630+0000 7fd5d1ec5640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5cc14e810 con 0x7fd5cc13f710 2024-03-25T18:29:40.634 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.630+0000 7fd5d1ec5640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5cc06dea0 con 0x7fd5cc1430c0 2024-03-25T18:29:40.634 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.630+0000 7fd5d1ec5640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:40.634 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.630+0000 7fd5d1ec5640 10 monclient(hunting): authenticate will time out at 7826.940525s 2024-03-25T18:29:40.634 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.632+0000 7fd5cb7fe640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fd5cc0a98f0 0x7fd5cc1340a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:42430/0 (socket says 172.21.15.155:42430) 2024-03-25T18:29:40.634 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.632+0000 7fd5cb7fe640 1 -- 172.21.15.155:0/3828626471 learned_addr learned my addr 172.21.15.155:0/3828626471 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:29:40.634 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.633+0000 7fd5b3fff640 10 client.?.objecter ms_handle_connect 0x7fd5cc0a98f0 2024-03-25T18:29:40.634 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.633+0000 7fd5b3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:40.643 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.642+0000 7fd5b3fff640 10 client.?.objecter ms_handle_connect 0x7fd5cc13f710 2024-03-25T18:29:40.643 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.642+0000 7fd5b3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:40.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.649+0000 7fd5b3fff640 1 -- 172.21.15.155:0/3828626471 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4126637250 0 0) 0x7fd5cc06fc80 con 0x7fd5cc0a98f0 2024-03-25T18:29:40.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.649+0000 7fd5b3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:29:40.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.649+0000 7fd5b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:40.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.649+0000 7fd5b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:40.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.649+0000 7fd5b3fff640 10 monclient(hunting): my global_id is 4446 2024-03-25T18:29:40.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.649+0000 7fd5b3fff640 1 -- 172.21.15.155:0/3828626471 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5b8004eb0 con 0x7fd5cc0a98f0 2024-03-25T18:29:40.653 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.651+0000 7fd5b3fff640 1 -- 172.21.15.155:0/3828626471 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 562627406 0 0) 0x7fd5cc14e810 con 0x7fd5cc13f710 2024-03-25T18:29:40.653 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.651+0000 7fd5b3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:29:40.653 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.651+0000 7fd5b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:40.653 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.651+0000 7fd5b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:40.653 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.651+0000 7fd5b3fff640 10 monclient(hunting): my global_id is 4438 2024-03-25T18:29:40.653 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.651+0000 7fd5b3fff640 1 -- 172.21.15.155:0/3828626471 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5cc06fc80 con 0x7fd5cc13f710 2024-03-25T18:29:40.658 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.657+0000 7fd5b3fff640 10 client.?.objecter ms_handle_connect 0x7fd5cc1430c0 2024-03-25T18:29:40.658 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.657+0000 7fd5b3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:40.667 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.666+0000 7fd5b3fff640 1 -- 172.21.15.155:0/3828626471 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3718018214 0 0) 0x7fd5cc06fc80 con 0x7fd5cc13f710 2024-03-25T18:29:40.667 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.666+0000 7fd5b3fff640 1 -- 172.21.15.155:0/3828626471 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd5cc14e810 con 0x7fd5cc13f710 2024-03-25T18:29:40.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.666+0000 7fd5b3fff640 1 -- 172.21.15.155:0/3828626471 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fd5c0001ae0 con 0x7fd5cc13f710 2024-03-25T18:29:40.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.666+0000 7fd5b3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:40.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.666+0000 7fd5b3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:29:40.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.666+0000 7fd5b3fff640 10 monclient(hunting): dump: 2024-03-25T18:29:40.668 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:29:40.668 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:29:40.668 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:40.668 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:40.668 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:40.668 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:29:40.668 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:29:40.668 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:29:40.668 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:29:40.668 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:29:40.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.666+0000 7fd5b3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:40.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.668+0000 7fd5b3fff640 1 -- 172.21.15.155:0/3828626471 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2103286988 0 0) 0x7fd5cc06dea0 con 0x7fd5cc1430c0 2024-03-25T18:29:40.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.668+0000 7fd5b3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:29:40.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.668+0000 7fd5b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:40.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.668+0000 7fd5b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:40.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.668+0000 7fd5b3fff640 10 monclient(hunting): my global_id is 4451 2024-03-25T18:29:40.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.668+0000 7fd5b3fff640 1 -- 172.21.15.155:0/3828626471 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5cc06fc80 con 0x7fd5cc1430c0 2024-03-25T18:29:40.678 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.677+0000 7fd5b3fff640 1 -- 172.21.15.155:0/3828626471 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 435673111 0 0) 0x7fd5cc06fc80 con 0x7fd5cc1430c0 2024-03-25T18:29:40.678 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.677+0000 7fd5b3fff640 1 -- 172.21.15.155:0/3828626471 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd5cc06dea0 con 0x7fd5cc1430c0 2024-03-25T18:29:40.678 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.677+0000 7fd5b3fff640 1 -- 172.21.15.155:0/3828626471 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fd5c4001bb0 con 0x7fd5cc1430c0 2024-03-25T18:29:40.678 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.677+0000 7fd5b3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:40.678 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.677+0000 7fd5b3fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:40.678 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.677+0000 7fd5b3fff640 10 monclient(hunting): dump: 2024-03-25T18:29:40.678 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:29:40.678 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:29:40.679 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:40.679 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:40.679 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:40.679 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:29:40.679 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:29:40.679 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:29:40.679 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:29:40.679 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:29:40.680 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.679+0000 7fd5b3fff640 1 -- 172.21.15.155:0/3828626471 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3652752777 0 0) 0x7fd5b8004eb0 con 0x7fd5cc0a98f0 2024-03-25T18:29:40.680 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.679+0000 7fd5b3fff640 1 -- 172.21.15.155:0/3828626471 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd5cc06fc80 con 0x7fd5cc0a98f0 2024-03-25T18:29:40.680 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.679+0000 7fd5b3fff640 1 -- 172.21.15.155:0/3828626471 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fd5bc001a10 con 0x7fd5cc0a98f0 2024-03-25T18:29:40.680 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.679+0000 7fd5b3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:40.680 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.679+0000 7fd5b3fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:40.680 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.679+0000 7fd5b3fff640 10 monclient(hunting): dump: 2024-03-25T18:29:40.680 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:29:40.680 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:29:40.680 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:40.680 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:40.681 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:40.681 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:29:40.681 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:29:40.681 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:29:40.681 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:29:40.681 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:29:40.685 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.684+0000 7fd5b3fff640 1 -- 172.21.15.155:0/3828626471 <== mon.1 v1:172.21.15.167:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2579353545 0 0) 0x7fd5cc14e810 con 0x7fd5cc13f710 2024-03-25T18:29:40.685 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.684+0000 7fd5b3fff640 1 -- 172.21.15.155:0/3828626471 >> v1:172.21.15.155:6790/0 conn(0x7fd5cc1430c0 legacy=0x7fd5cc1df530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:40.685 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.684+0000 7fd5b3fff640 1 -- 172.21.15.155:0/3828626471 >> v1:172.21.15.155:6789/0 conn(0x7fd5cc0a98f0 legacy=0x7fd5cc1340a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:40.685 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.684+0000 7fd5b3fff640 10 monclient: _finish_hunting 0 2024-03-25T18:29:40.685 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.684+0000 7fd5b3fff640 1 monclient: found mon.b 2024-03-25T18:29:40.685 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.684+0000 7fd5b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:40.685 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.684+0000 7fd5b3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:29:40.686 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.684+0000 7fd5b3fff640 1 -- 172.21.15.155:0/3828626471 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5cc06e030 con 0x7fd5cc13f710 2024-03-25T18:29:40.686 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.684+0000 7fd5b3fff640 10 monclient: _finish_auth 0 2024-03-25T18:29:40.686 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.684+0000 7fd5b3fff640 10 monclient: _check_auth_tickets 2024-03-25T18:29:40.686 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.684+0000 7fd5b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:40.686 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.684+0000 7fd5d1ec5640 5 monclient: authenticate success, global_id 4438 2024-03-25T18:29:40.686 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.684+0000 7fd5d1ec5640 10 monclient: _renew_subs 2024-03-25T18:29:40.686 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.684+0000 7fd5d1ec5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:29:40.686 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.684+0000 7fd5d1ec5640 1 -- 172.21.15.155:0/3828626471 --> v1:172.21.15.167:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5cc1347b0 con 0x7fd5cc13f710 2024-03-25T18:29:40.686 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.684+0000 7fd5d1ec5640 10 client.4438.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:40.686 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.684+0000 7fd5d1ec5640 10 monclient: _renew_subs 2024-03-25T18:29:40.686 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.684+0000 7fd5d1ec5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:29:40.686 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.684+0000 7fd5d1ec5640 1 -- 172.21.15.155:0/3828626471 --> v1:172.21.15.167:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5cc14c010 con 0x7fd5cc13f710 2024-03-25T18:29:40.686 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.684+0000 7fd5d1ec5640 1 librados: init done 2024-03-25T18:29:40.687 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.685+0000 7fd5d1ec5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:40.687 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.685+0000 7fd5d1ec5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:40.687 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.685+0000 7fd5d1ec5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:29:40.687 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.685+0000 7fd5d1ec5640 1 -- 172.21.15.155:0/3828626471 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5cc0627b0 con 0x7fd5cc13f710 2024-03-25T18:29:40.972 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.970+0000 7fd5b3fff640 1 -- 172.21.15.155:0/3828626471 <== mon.1 v1:172.21.15.167:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd5c00038a0 con 0x7fd5cc13f710 2024-03-25T18:29:40.972 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.970+0000 7fd5b3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:40.972 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.970+0000 7fd5b3fff640 1 -- 172.21.15.155:0/3828626471 <== mon.1 v1:172.21.15.167:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fd5c0003bb0 con 0x7fd5cc13f710 2024-03-25T18:29:40.972 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.970+0000 7fd5b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:40.972 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.970+0000 7fd5b3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:40.972 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.970+0000 7fd5b3fff640 10 monclient: dump: 2024-03-25T18:29:40.972 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:29:40.972 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:29:40.972 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:40.972 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:40.972 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:40.973 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:29:40.973 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:29:40.973 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:29:40.973 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:29:40.973 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:29:40.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.970+0000 7fd5c9ffb640 4 set_mon_vals no callback set 2024-03-25T18:29:40.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.971+0000 7fd5b3fff640 1 -- 172.21.15.155:0/3828626471 <== mon.1 v1:172.21.15.167:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7fd5c0019b80 con 0x7fd5cc13f710 2024-03-25T18:29:40.974 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.972+0000 7fd5cb7fe640 10 monclient: get_auth_request con 0x7fd5b806e110 auth_method 0 2024-03-25T18:29:40.974 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.972+0000 7fd5b3fff640 10 client.4438.objecter ms_handle_connect 0x7fd5b806e110 2024-03-25T18:29:40.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.977+0000 7fd5b3fff640 1 -- 172.21.15.155:0/3828626471 <== mon.1 v1:172.21.15.167:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7fd5c00536d0 con 0x7fd5cc13f710 2024-03-25T18:29:40.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.977+0000 7fd5b3fff640 10 client.4438.objecter ms_dispatch 0x7fd5cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:40.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.977+0000 7fd5b3fff640 3 client.4438.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:40.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.977+0000 7fd5b3fff640 3 client.4438.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:40.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.977+0000 7fd5b3fff640 20 client.4438.objecter dump_active .. 0 homeless 2024-03-25T18:29:40.981 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.980+0000 7fd5b3fff640 1 -- 172.21.15.155:0/3828626471 <== mon.1 v1:172.21.15.167:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd5c004f1e0 con 0x7fd5cc13f710 2024-03-25T18:29:40.981 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.980+0000 7fd5b3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:40.981 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:40.980+0000 7fd5b3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:41.227 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:41.226+0000 7fd5d1ec5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:41.227 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:41.226+0000 7fd5d1ec5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:41.228 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:41.226+0000 7fd5d1ec5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:29:41.228 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:41.226+0000 7fd5d1ec5640 1 -- 172.21.15.155:0/3828626471 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd5cc1419e0 con 0x7fd5cc13f710 2024-03-25T18:29:41.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:41.306+0000 7fd5b3fff640 1 -- 172.21.15.155:0/3828626471 <== mon.1 v1:172.21.15.167:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7fd5c0051110 con 0x7fd5cc13f710 2024-03-25T18:29:41.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:41.306+0000 7fd5b3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:41.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:41.306+0000 7fd5b3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:41.308 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:29:41.308 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:41.310 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:41.308+0000 7fd5d1ec5640 10 librados: watch_flush enter 2024-03-25T18:29:41.310 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:41.308+0000 7fd5d1ec5640 10 librados: watch_flush exit 2024-03-25T18:29:41.310 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:41.308+0000 7fd5d1ec5640 20 client.4438.objecter shutdown clearing up homeless session... 2024-03-25T18:29:41.310 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:41.308+0000 7fd5d1ec5640 10 client.4438.objecter successfully canceled tick 2024-03-25T18:29:41.310 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:41.308+0000 7fd5d1ec5640 1 -- 172.21.15.155:0/3828626471 >> v1:172.21.15.167:6800/2919070503 conn(0x7fd5b806e110 legacy=0x7fd5b80705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:41.310 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:41.308+0000 7fd5d1ec5640 10 monclient: shutdown 2024-03-25T18:29:41.310 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:41.308+0000 7fd5d1ec5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:41.310 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:41.308+0000 7fd5d1ec5640 1 -- 172.21.15.155:0/3828626471 >> v1:172.21.15.167:6789/0 conn(0x7fd5cc13f710 legacy=0x7fd5cc1dee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:41.311 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:41.309+0000 7fd5d1ec5640 1 -- 172.21.15.155:0/3828626471 shutdown_connections 2024-03-25T18:29:41.311 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:41.309+0000 7fd5d1ec5640 1 -- 172.21.15.155:0/3828626471 >> 172.21.15.155:0/3828626471 conn(0x7fd5cc0b7ea0 msgr2=0x7fd5cc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:41.311 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:41.309+0000 7fd5d1ec5640 1 -- 172.21.15.155:0/3828626471 shutdown_connections 2024-03-25T18:29:41.311 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:41.309+0000 7fd5d1ec5640 1 -- 172.21.15.155:0/3828626471 wait complete. 2024-03-25T18:29:41.311 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:41.309+0000 7fd5d1ec5640 1 librados: shutdown 2024-03-25T18:29:41.322 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:46.237 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:05:19.841 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:46.285 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:05:19.890 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:46.317 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:05:19.921 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:46.335 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:05:19.940 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:46.339 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:05:19.918 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:46.342 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:05:19.947 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:46.345 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:05:19.923 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:46.382 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:05:19.961 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:46.391 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:05:19.995 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:48.130 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:48.250 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.249+0000 7f000a80e640 10 monclient: get_monmap_and_config 2024-03-25T18:29:48.251 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.249+0000 7f000a80e640 10 monclient: build_initial_monmap 2024-03-25T18:29:48.251 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.249+0000 7f000a80e640 10 monclient: monmap: 2024-03-25T18:29:48.251 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:29:48.251 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:29:48.251 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:29:48.250596+0000 2024-03-25T18:29:48.251 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:29:48.250596+0000 2024-03-25T18:29:48.251 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:48.251 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:29:48.251 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:29:48.251 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:29:48.251 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:29:48.251 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:29:48.253 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.250+0000 7f000a80e640 1 Processor -- start 2024-03-25T18:29:48.253 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.250+0000 7f000a80e640 1 -- start start 2024-03-25T18:29:48.253 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.250+0000 7f000a80e640 10 monclient: init 2024-03-25T18:29:48.253 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.251+0000 7f000a80e640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:48.253 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.251+0000 7f000a80e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:29:48.253 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.251+0000 7f000a80e640 10 monclient(hunting): picked mon.noname-a con 0x7f00040a98f0 addr v1:172.21.15.155:6789/0 2024-03-25T18:29:48.253 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.251+0000 7f000a80e640 10 monclient(hunting): picked mon.noname-b con 0x7f000413f710 addr v1:172.21.15.155:6790/0 2024-03-25T18:29:48.253 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.251+0000 7f000a80e640 10 monclient(hunting): picked mon.noname-c con 0x7f00041430c0 addr v1:172.21.15.167:6789/0 2024-03-25T18:29:48.253 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.251+0000 7f000a80e640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0004142520 con 0x7f00040a98f0 2024-03-25T18:29:48.253 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.251+0000 7f000a80e640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00040b28a0 con 0x7f00041430c0 2024-03-25T18:29:48.253 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.251+0000 7f000a80e640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00040b2a30 con 0x7f000413f710 2024-03-25T18:29:48.254 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.251+0000 7f000a80e640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:48.254 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.251+0000 7f000a80e640 10 monclient(hunting): authenticate will time out at 7834.561167s 2024-03-25T18:29:48.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.260+0000 7f0008dab640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f00041430c0 0x7f000414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:42654/0 (socket says 172.21.15.155:42654) 2024-03-25T18:29:48.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.260+0000 7f0008dab640 1 -- 172.21.15.155:0/1542099022 learned_addr learned my addr 172.21.15.155:0/1542099022 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:29:48.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.275+0000 7f00037fe640 1 -- 172.21.15.155:0/1542099022 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3775731510 0 0) 0x7f0004142520 con 0x7f00040a98f0 2024-03-25T18:29:48.277 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.275+0000 7f00037fe640 10 monclient(hunting): _negotiate 2024-03-25T18:29:48.277 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.275+0000 7f00037fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:48.277 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.275+0000 7f00037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:48.277 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.275+0000 7f00037fe640 10 monclient(hunting): my global_id is 4449 2024-03-25T18:29:48.277 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.275+0000 7f00037fe640 1 -- 172.21.15.155:0/1542099022 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efff0003520 con 0x7f00040a98f0 2024-03-25T18:29:48.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.278+0000 7f00037fe640 1 -- 172.21.15.155:0/1542099022 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 559991423 0 0) 0x7f00040b2a30 con 0x7f000413f710 2024-03-25T18:29:48.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.278+0000 7f00037fe640 10 monclient(hunting): _negotiate 2024-03-25T18:29:48.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.278+0000 7f00037fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:48.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.278+0000 7f00037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:48.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.278+0000 7f00037fe640 10 monclient(hunting): my global_id is 4454 2024-03-25T18:29:48.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.278+0000 7f00037fe640 1 -- 172.21.15.155:0/1542099022 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0004142520 con 0x7f000413f710 2024-03-25T18:29:48.282 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.281+0000 7f00037fe640 1 -- 172.21.15.155:0/1542099022 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1726382832 0 0) 0x7f00040b28a0 con 0x7f00041430c0 2024-03-25T18:29:48.282 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.281+0000 7f00037fe640 10 monclient(hunting): _negotiate 2024-03-25T18:29:48.282 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.281+0000 7f00037fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:48.282 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.281+0000 7f00037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:48.282 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.281+0000 7f00037fe640 10 monclient(hunting): my global_id is 4441 2024-03-25T18:29:48.282 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.281+0000 7f00037fe640 1 -- 172.21.15.155:0/1542099022 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00040b2a30 con 0x7f00041430c0 2024-03-25T18:29:48.289 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.287+0000 7f00037fe640 1 -- 172.21.15.155:0/1542099022 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 527971323 0 0) 0x7f0004142520 con 0x7f000413f710 2024-03-25T18:29:48.289 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.287+0000 7f00037fe640 1 -- 172.21.15.155:0/1542099022 >> v1:172.21.15.167:6789/0 conn(0x7f00041430c0 legacy=0x7f000414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:48.289 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.287+0000 7f00037fe640 1 -- 172.21.15.155:0/1542099022 >> v1:172.21.15.155:6789/0 conn(0x7f00040a98f0 legacy=0x7f00040a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:48.289 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.287+0000 7f00037fe640 10 monclient: _finish_hunting 0 2024-03-25T18:29:48.289 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.287+0000 7f00037fe640 1 monclient: found mon.noname-b 2024-03-25T18:29:48.289 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.287+0000 7f00037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:48.289 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.287+0000 7f00037fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-25T18:29:48.289 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.287+0000 7f00037fe640 1 -- 172.21.15.155:0/1542099022 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00040b44c0 con 0x7f000413f710 2024-03-25T18:29:48.289 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.287+0000 7f00037fe640 10 monclient: _finish_auth 0 2024-03-25T18:29:48.289 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.287+0000 7f00037fe640 10 monclient: _check_auth_tickets 2024-03-25T18:29:48.289 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.287+0000 7f00037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:48.290 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.287+0000 7f00037fe640 1 -- 172.21.15.155:0/1542099022 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7effec001670 con 0x7f000413f710 2024-03-25T18:29:48.290 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.287+0000 7f00037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:48.290 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.287+0000 7f00037fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:29:48.290 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.287+0000 7f00037fe640 10 monclient: dump: 2024-03-25T18:29:48.290 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:29:48.290 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:29:48.290 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:48.290 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:48.290 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:48.290 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:29:48.290 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:29:48.290 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:29:48.290 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:29:48.290 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:29:48.291 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.287+0000 7f000a80e640 5 monclient: authenticate success, global_id 4454 2024-03-25T18:29:48.291 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.287+0000 7f000a80e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:48.305 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.304+0000 7f00037fe640 1 -- 172.21.15.155:0/1542099022 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7effec0017d0 con 0x7f000413f710 2024-03-25T18:29:48.305 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.304+0000 7f00037fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:48.305 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.304+0000 7f00037fe640 1 -- 172.21.15.155:0/1542099022 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7effec001be0 con 0x7f000413f710 2024-03-25T18:29:48.305 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.304+0000 7f000a80e640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:48.305 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.304+0000 7f000a80e640 4 set_mon_vals no callback set 2024-03-25T18:29:48.306 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.304+0000 7f00037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:48.306 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.304+0000 7f00037fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:48.306 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.304+0000 7f00037fe640 10 monclient: dump: 2024-03-25T18:29:48.306 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:29:48.306 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:29:48.306 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:48.306 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:48.306 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:48.306 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:29:48.306 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:29:48.306 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:29:48.306 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:29:48.306 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:29:48.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.305+0000 7f000a80e640 10 monclient: shutdown 2024-03-25T18:29:48.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.305+0000 7f000a80e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:48.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.305+0000 7f000a80e640 1 -- 172.21.15.155:0/1542099022 >> v1:172.21.15.155:6790/0 conn(0x7f000413f710 legacy=0x7f00041429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:48.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.305+0000 7f000a80e640 1 -- 172.21.15.155:0/1542099022 shutdown_connections 2024-03-25T18:29:48.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.305+0000 7f000a80e640 1 -- 172.21.15.155:0/1542099022 >> 172.21.15.155:0/1542099022 conn(0x7f00040b7ea0 msgr2=0x7f00040b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:48.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.305+0000 7f000a80e640 1 -- 172.21.15.155:0/1542099022 shutdown_connections 2024-03-25T18:29:48.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.305+0000 7f000a80e640 1 -- 172.21.15.155:0/1542099022 wait complete. 2024-03-25T18:29:48.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.306+0000 7f000a80e640 10 monclient: build_initial_monmap 2024-03-25T18:29:48.309 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.306+0000 7f000a80e640 10 monclient: monmap: 2024-03-25T18:29:48.309 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:29:48.309 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:48.309 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:29:48.309 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:29:48.309 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:48.309 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:29:48.309 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:29:48.309 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:29:48.309 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:29:48.309 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:29:48.309 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.306+0000 7f000a80e640 1 librados: starting msgr at 2024-03-25T18:29:48.309 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.306+0000 7f000a80e640 1 librados: starting objecter 2024-03-25T18:29:48.309 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.307+0000 7f000a80e640 1 Processor -- start 2024-03-25T18:29:48.310 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.307+0000 7f000a80e640 1 -- start start 2024-03-25T18:29:48.310 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.307+0000 7f000a80e640 1 librados: setting wanted keys 2024-03-25T18:29:48.310 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.307+0000 7f000a80e640 1 librados: calling monclient init 2024-03-25T18:29:48.310 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.307+0000 7f000a80e640 10 monclient: init 2024-03-25T18:29:48.311 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.307+0000 7f000a80e640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:48.311 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.307+0000 7f000a80e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:29:48.311 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.307+0000 7f000a80e640 10 monclient(hunting): picked mon.noname-b con 0x7f00040a98f0 addr v1:172.21.15.167:6789/0 2024-03-25T18:29:48.311 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.307+0000 7f000a80e640 10 monclient(hunting): picked mon.noname-a con 0x7f000413f710 addr v1:172.21.15.155:6789/0 2024-03-25T18:29:48.311 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.307+0000 7f000a80e640 10 monclient(hunting): picked mon.noname-c con 0x7f00041430c0 addr v1:172.21.15.155:6790/0 2024-03-25T18:29:48.311 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.307+0000 7f000a80e640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0004138700 con 0x7f000413f710 2024-03-25T18:29:48.311 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.307+0000 7f000a80e640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0004138b90 con 0x7f00040a98f0 2024-03-25T18:29:48.311 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.307+0000 7f000a80e640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00040b16d0 con 0x7f00041430c0 2024-03-25T18:29:48.312 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.307+0000 7f000a80e640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:48.312 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.308+0000 7f000a80e640 10 monclient(hunting): authenticate will time out at 7834.617730s 2024-03-25T18:29:48.312 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.311+0000 7efffbfff640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f000413f710 0x7f00041da880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:58930/0 (socket says 172.21.15.155:58930) 2024-03-25T18:29:48.312 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.311+0000 7efffbfff640 1 -- 172.21.15.155:0/3954212549 learned_addr learned my addr 172.21.15.155:0/3954212549 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:29:48.312 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.312+0000 7f00017fa640 10 client.?.objecter ms_handle_connect 0x7f000413f710 2024-03-25T18:29:48.313 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.312+0000 7f00017fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:48.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.321+0000 7f00017fa640 10 client.?.objecter ms_handle_connect 0x7f00041430c0 2024-03-25T18:29:48.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.321+0000 7f00017fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:48.327 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.325+0000 7f00017fa640 1 -- 172.21.15.155:0/3954212549 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4073716439 0 0) 0x7f0004138700 con 0x7f000413f710 2024-03-25T18:29:48.328 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.325+0000 7f00017fa640 10 monclient(hunting): _negotiate 2024-03-25T18:29:48.328 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.326+0000 7f00017fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:48.328 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.326+0000 7f00017fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:48.328 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.326+0000 7f00017fa640 10 monclient(hunting): my global_id is 4452 2024-03-25T18:29:48.328 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.326+0000 7f00017fa640 1 -- 172.21.15.155:0/3954212549 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effe4004eb0 con 0x7f000413f710 2024-03-25T18:29:48.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.328+0000 7f00017fa640 1 -- 172.21.15.155:0/3954212549 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2173671874 0 0) 0x7f00040b16d0 con 0x7f00041430c0 2024-03-25T18:29:48.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.329+0000 7f00017fa640 10 monclient(hunting): _negotiate 2024-03-25T18:29:48.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.329+0000 7f00017fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:48.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.329+0000 7f00017fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:48.331 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.329+0000 7f00017fa640 10 monclient(hunting): my global_id is 4457 2024-03-25T18:29:48.331 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.329+0000 7f00017fa640 1 -- 172.21.15.155:0/3954212549 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0004138700 con 0x7f00041430c0 2024-03-25T18:29:48.331 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.330+0000 7f00017fa640 10 client.?.objecter ms_handle_connect 0x7f00040a98f0 2024-03-25T18:29:48.331 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.330+0000 7f00017fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:48.339 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.337+0000 7f00017fa640 1 -- 172.21.15.155:0/3954212549 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 951408774 0 0) 0x7f0004138700 con 0x7f00041430c0 2024-03-25T18:29:48.339 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.338+0000 7f00017fa640 1 -- 172.21.15.155:0/3954212549 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f00040b16d0 con 0x7f00041430c0 2024-03-25T18:29:48.339 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.338+0000 7f00017fa640 1 -- 172.21.15.155:0/3954212549 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7efffc001bd0 con 0x7f00041430c0 2024-03-25T18:29:48.339 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.338+0000 7f00017fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:48.339 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.338+0000 7f00017fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:29:48.339 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.338+0000 7f00017fa640 10 monclient(hunting): dump: 2024-03-25T18:29:48.339 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:29:48.339 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:29:48.340 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:48.340 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:48.340 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:48.340 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:29:48.340 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:29:48.340 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:29:48.340 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:29:48.340 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:29:48.340 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.338+0000 7f00017fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:48.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.341+0000 7f00017fa640 1 -- 172.21.15.155:0/3954212549 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2986480708 0 0) 0x7effe4004eb0 con 0x7f000413f710 2024-03-25T18:29:48.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.341+0000 7f00017fa640 1 -- 172.21.15.155:0/3954212549 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0004138700 con 0x7f000413f710 2024-03-25T18:29:48.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.341+0000 7f00017fa640 1 -- 172.21.15.155:0/3954212549 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7effec001670 con 0x7f000413f710 2024-03-25T18:29:48.343 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.341+0000 7f00017fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:48.343 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.341+0000 7f00017fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:48.343 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.341+0000 7f00017fa640 10 monclient(hunting): dump: 2024-03-25T18:29:48.343 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:29:48.343 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:29:48.343 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:48.343 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:48.343 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:48.343 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:29:48.343 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:29:48.343 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:29:48.343 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:29:48.343 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:29:48.344 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.343+0000 7f00017fa640 1 -- 172.21.15.155:0/3954212549 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 520125111 0 0) 0x7f0004138b90 con 0x7f00040a98f0 2024-03-25T18:29:48.344 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.343+0000 7f00017fa640 10 monclient(hunting): _negotiate 2024-03-25T18:29:48.344 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.343+0000 7f00017fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:48.344 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.343+0000 7f00017fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:48.344 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.343+0000 7f00017fa640 10 monclient(hunting): my global_id is 4444 2024-03-25T18:29:48.344 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.343+0000 7f00017fa640 1 -- 172.21.15.155:0/3954212549 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effe4004eb0 con 0x7f00040a98f0 2024-03-25T18:29:48.357 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.355+0000 7f00017fa640 1 -- 172.21.15.155:0/3954212549 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1113067065 0 0) 0x7f0004138700 con 0x7f000413f710 2024-03-25T18:29:48.357 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.355+0000 7f00017fa640 1 -- 172.21.15.155:0/3954212549 >> v1:172.21.15.155:6790/0 conn(0x7f00041430c0 legacy=0x7f00041e1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:48.357 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.355+0000 7f00017fa640 1 -- 172.21.15.155:0/3954212549 >> v1:172.21.15.167:6789/0 conn(0x7f00040a98f0 legacy=0x7f0004137820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:48.357 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.355+0000 7f00017fa640 10 monclient: _finish_hunting 0 2024-03-25T18:29:48.357 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.355+0000 7f00017fa640 1 monclient: found mon.a 2024-03-25T18:29:48.357 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.355+0000 7f00017fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:48.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.355+0000 7f00017fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:29:48.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.355+0000 7f00017fa640 1 -- 172.21.15.155:0/3954212549 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0004138050 con 0x7f000413f710 2024-03-25T18:29:48.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.355+0000 7f00017fa640 10 monclient: _finish_auth 0 2024-03-25T18:29:48.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.355+0000 7f00017fa640 10 monclient: _check_auth_tickets 2024-03-25T18:29:48.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.355+0000 7f00017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:48.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.355+0000 7f000a80e640 5 monclient: authenticate success, global_id 4452 2024-03-25T18:29:48.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.355+0000 7f000a80e640 10 monclient: _renew_subs 2024-03-25T18:29:48.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.355+0000 7f000a80e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:29:48.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.355+0000 7f000a80e640 1 -- 172.21.15.155:0/3954212549 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00041382a0 con 0x7f000413f710 2024-03-25T18:29:48.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.355+0000 7f000a80e640 10 client.4452.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:48.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.355+0000 7f000a80e640 10 monclient: _renew_subs 2024-03-25T18:29:48.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.355+0000 7f000a80e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:29:48.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.355+0000 7f000a80e640 1 -- 172.21.15.155:0/3954212549 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f00041db170 con 0x7f000413f710 2024-03-25T18:29:48.359 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.355+0000 7f000a80e640 1 librados: init done 2024-03-25T18:29:48.359 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.357+0000 7f000a80e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:48.359 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.357+0000 7f000a80e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:48.359 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.357+0000 7f000a80e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:29:48.359 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.357+0000 7f000a80e640 1 -- 172.21.15.155:0/3954212549 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00041db420 con 0x7f000413f710 2024-03-25T18:29:48.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.661+0000 7f00017fa640 1 -- 172.21.15.155:0/3954212549 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7effec0038a0 con 0x7f000413f710 2024-03-25T18:29:48.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.661+0000 7f00017fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:48.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.661+0000 7f00017fa640 1 -- 172.21.15.155:0/3954212549 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7effec003cf0 con 0x7f000413f710 2024-03-25T18:29:48.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.661+0000 7f00017fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:48.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.661+0000 7f00017fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:48.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.661+0000 7f00017fa640 10 monclient: dump: 2024-03-25T18:29:48.662 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:29:48.663 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:29:48.663 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:48.663 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:48.663 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:48.663 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:29:48.663 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:29:48.663 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:29:48.663 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:29:48.663 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:29:48.663 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.661+0000 7f0002ffd640 4 set_mon_vals no callback set 2024-03-25T18:29:48.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.662+0000 7f00017fa640 1 -- 172.21.15.155:0/3954212549 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7effec003f10 con 0x7f000413f710 2024-03-25T18:29:48.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.662+0000 7f0003fff640 10 monclient: get_auth_request con 0x7effe406df30 auth_method 0 2024-03-25T18:29:48.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.662+0000 7f00017fa640 10 client.4452.objecter ms_handle_connect 0x7effe406df30 2024-03-25T18:29:48.670 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.668+0000 7f00017fa640 1 -- 172.21.15.155:0/3954212549 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7effec0549b0 con 0x7f000413f710 2024-03-25T18:29:48.670 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.668+0000 7f00017fa640 10 client.4452.objecter ms_dispatch 0x7f0004000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:48.670 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.668+0000 7f00017fa640 3 client.4452.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:48.670 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.668+0000 7f00017fa640 3 client.4452.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:48.670 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.668+0000 7f00017fa640 20 client.4452.objecter dump_active .. 0 homeless 2024-03-25T18:29:48.670 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.668+0000 7f00017fa640 1 -- 172.21.15.155:0/3954212549 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7effec0504c0 con 0x7f000413f710 2024-03-25T18:29:48.670 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.668+0000 7f00017fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:48.670 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.668+0000 7f00017fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:48.920 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.919+0000 7f000a80e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:48.920 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.919+0000 7f000a80e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:48.920 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.919+0000 7f000a80e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:29:48.920 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.919+0000 7f000a80e640 1 -- 172.21.15.155:0/3954212549 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f00040b3ef0 con 0x7f000413f710 2024-03-25T18:29:48.957 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.955+0000 7f00017fa640 1 -- 172.21.15.155:0/3954212549 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7effec0523f0 con 0x7f000413f710 2024-03-25T18:29:48.957 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.955+0000 7f00017fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:48.957 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.955+0000 7f00017fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:48.957 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:29:48.957 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:48.959 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.958+0000 7f000a80e640 10 librados: watch_flush enter 2024-03-25T18:29:48.959 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.958+0000 7f000a80e640 10 librados: watch_flush exit 2024-03-25T18:29:48.960 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.958+0000 7f000a80e640 20 client.4452.objecter shutdown clearing up homeless session... 2024-03-25T18:29:48.960 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.958+0000 7f000a80e640 10 client.4452.objecter successfully canceled tick 2024-03-25T18:29:48.960 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.958+0000 7f000a80e640 1 -- 172.21.15.155:0/3954212549 >> v1:172.21.15.167:6800/2919070503 conn(0x7effe406df30 legacy=0x7effe40703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:48.960 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.958+0000 7f000a80e640 10 monclient: shutdown 2024-03-25T18:29:48.960 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.958+0000 7f000a80e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:48.960 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.958+0000 7f000a80e640 1 -- 172.21.15.155:0/3954212549 >> v1:172.21.15.155:6789/0 conn(0x7f000413f710 legacy=0x7f00041da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:48.960 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.958+0000 7f000a80e640 1 -- 172.21.15.155:0/3954212549 shutdown_connections 2024-03-25T18:29:48.960 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.958+0000 7f000a80e640 1 -- 172.21.15.155:0/3954212549 >> 172.21.15.155:0/3954212549 conn(0x7f00040b7ea0 msgr2=0x7f0004074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:48.960 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.958+0000 7f000a80e640 1 -- 172.21.15.155:0/3954212549 shutdown_connections 2024-03-25T18:29:48.960 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.958+0000 7f000a80e640 1 -- 172.21.15.155:0/3954212549 wait complete. 2024-03-25T18:29:48.960 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:48.958+0000 7f000a80e640 1 librados: shutdown 2024-03-25T18:29:48.972 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:55.779 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:55.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.892+0000 7f000bf8d640 10 monclient: get_monmap_and_config 2024-03-25T18:29:55.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.892+0000 7f000bf8d640 10 monclient: build_initial_monmap 2024-03-25T18:29:55.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.892+0000 7f000bf8d640 10 monclient: monmap: 2024-03-25T18:29:55.894 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:29:55.894 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:29:55.894 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:29:55.893206+0000 2024-03-25T18:29:55.894 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:29:55.893206+0000 2024-03-25T18:29:55.894 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:55.894 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:29:55.894 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:29:55.894 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:29:55.895 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:29:55.895 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:29:55.896 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.892+0000 7f000bf8d640 1 Processor -- start 2024-03-25T18:29:55.896 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.893+0000 7f000bf8d640 1 -- start start 2024-03-25T18:29:55.896 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.893+0000 7f000bf8d640 10 monclient: init 2024-03-25T18:29:55.896 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.893+0000 7f000bf8d640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:55.896 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.893+0000 7f000bf8d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:29:55.896 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.893+0000 7f000bf8d640 10 monclient(hunting): picked mon.noname-c con 0x7f00040a98f0 addr v1:172.21.15.167:6789/0 2024-03-25T18:29:55.896 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.893+0000 7f000bf8d640 10 monclient(hunting): picked mon.noname-a con 0x7f000413f710 addr v1:172.21.15.155:6789/0 2024-03-25T18:29:55.896 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.893+0000 7f000bf8d640 10 monclient(hunting): picked mon.noname-b con 0x7f00041430c0 addr v1:172.21.15.155:6790/0 2024-03-25T18:29:55.896 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.893+0000 7f000bf8d640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0004142520 con 0x7f000413f710 2024-03-25T18:29:55.896 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.893+0000 7f000bf8d640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00040b28a0 con 0x7f00040a98f0 2024-03-25T18:29:55.897 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.893+0000 7f000bf8d640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00040b2a30 con 0x7f00041430c0 2024-03-25T18:29:55.897 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.893+0000 7f000bf8d640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:55.897 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.893+0000 7f000bf8d640 10 monclient(hunting): authenticate will time out at 7842.203713s 2024-03-25T18:29:55.909 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.908+0000 7f0009d29640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f00040a98f0 0x7f00040a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:47438/0 (socket says 172.21.15.155:47438) 2024-03-25T18:29:55.910 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.908+0000 7f0009d29640 1 -- 172.21.15.155:0/2989807869 learned_addr learned my addr 172.21.15.155:0/2989807869 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:29:55.919 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.918+0000 7f0008d27640 1 -- 172.21.15.155:0/2989807869 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2915748916 0 0) 0x7f0004142520 con 0x7f000413f710 2024-03-25T18:29:55.919 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.918+0000 7f0008d27640 10 monclient(hunting): _negotiate 2024-03-25T18:29:55.919 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.918+0000 7f0008d27640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:55.919 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.918+0000 7f0008d27640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:55.919 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.918+0000 7f0008d27640 10 monclient(hunting): my global_id is 4455 2024-03-25T18:29:55.919 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.918+0000 7f0008d27640 1 -- 172.21.15.155:0/2989807869 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efff0003520 con 0x7f000413f710 2024-03-25T18:29:55.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.923+0000 7f0008d27640 1 -- 172.21.15.155:0/2989807869 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4278661037 0 0) 0x7f00040b2a30 con 0x7f00041430c0 2024-03-25T18:29:55.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.923+0000 7f0008d27640 10 monclient(hunting): _negotiate 2024-03-25T18:29:55.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.923+0000 7f0008d27640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:55.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.923+0000 7f0008d27640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:55.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.923+0000 7f0008d27640 10 monclient(hunting): my global_id is 4460 2024-03-25T18:29:55.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.923+0000 7f0008d27640 1 -- 172.21.15.155:0/2989807869 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0004142520 con 0x7f00041430c0 2024-03-25T18:29:55.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.924+0000 7f0008d27640 1 -- 172.21.15.155:0/2989807869 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3563441179 0 0) 0x7f00040b28a0 con 0x7f00040a98f0 2024-03-25T18:29:55.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.924+0000 7f0008d27640 10 monclient(hunting): _negotiate 2024-03-25T18:29:55.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.924+0000 7f0008d27640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:55.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.924+0000 7f0008d27640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:55.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.924+0000 7f0008d27640 10 monclient(hunting): my global_id is 4447 2024-03-25T18:29:55.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.924+0000 7f0008d27640 1 -- 172.21.15.155:0/2989807869 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00040b2a30 con 0x7f00040a98f0 2024-03-25T18:29:55.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.925+0000 7f0008d27640 1 -- 172.21.15.155:0/2989807869 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1470467751 0 0) 0x7efff0003520 con 0x7f000413f710 2024-03-25T18:29:55.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.925+0000 7f0008d27640 1 -- 172.21.15.155:0/2989807869 >> v1:172.21.15.155:6790/0 conn(0x7f00041430c0 legacy=0x7f000414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:55.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.925+0000 7f0008d27640 1 -- 172.21.15.155:0/2989807869 >> v1:172.21.15.167:6789/0 conn(0x7f00040a98f0 legacy=0x7f00040a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:55.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.925+0000 7f0008d27640 10 monclient: _finish_hunting 0 2024-03-25T18:29:55.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.925+0000 7f0008d27640 1 monclient: found mon.noname-a 2024-03-25T18:29:55.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.925+0000 7f0008d27640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:55.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.925+0000 7f0008d27640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-25T18:29:55.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.925+0000 7f0008d27640 1 -- 172.21.15.155:0/2989807869 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00040b44c0 con 0x7f000413f710 2024-03-25T18:29:55.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.925+0000 7f0008d27640 10 monclient: _finish_auth 0 2024-03-25T18:29:55.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.925+0000 7f0008d27640 10 monclient: _check_auth_tickets 2024-03-25T18:29:55.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.925+0000 7f0008d27640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:55.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.925+0000 7f0008d27640 1 -- 172.21.15.155:0/2989807869 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7efff4001690 con 0x7f000413f710 2024-03-25T18:29:55.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.925+0000 7f0008d27640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:55.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.925+0000 7f0008d27640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:29:55.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.925+0000 7f0008d27640 10 monclient: dump: 2024-03-25T18:29:55.928 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:29:55.928 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:29:55.928 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:55.928 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:55.928 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:55.928 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:29:55.928 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:29:55.928 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:29:55.928 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:29:55.928 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:29:55.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.925+0000 7f000bf8d640 5 monclient: authenticate success, global_id 4455 2024-03-25T18:29:55.929 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.925+0000 7f000bf8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:55.934 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.933+0000 7f0008d27640 1 -- 172.21.15.155:0/2989807869 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efff40017f0 con 0x7f000413f710 2024-03-25T18:29:55.934 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.933+0000 7f0008d27640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:55.934 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.933+0000 7f0008d27640 1 -- 172.21.15.155:0/2989807869 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7efff4001c00 con 0x7f000413f710 2024-03-25T18:29:55.934 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.933+0000 7f000bf8d640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:55.934 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.933+0000 7f000bf8d640 4 set_mon_vals no callback set 2024-03-25T18:29:55.934 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.933+0000 7f0008d27640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:55.935 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.933+0000 7f0008d27640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:55.935 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.933+0000 7f0008d27640 10 monclient: dump: 2024-03-25T18:29:55.935 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:29:55.935 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:29:55.935 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:55.935 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:55.935 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:55.935 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:29:55.935 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:29:55.935 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:29:55.936 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:29:55.936 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:29:55.937 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.934+0000 7f000bf8d640 10 monclient: shutdown 2024-03-25T18:29:55.937 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.934+0000 7f000bf8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:55.937 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.934+0000 7f000bf8d640 1 -- 172.21.15.155:0/2989807869 >> v1:172.21.15.155:6789/0 conn(0x7f000413f710 legacy=0x7f00041429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:55.938 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.934+0000 7f000bf8d640 1 -- 172.21.15.155:0/2989807869 shutdown_connections 2024-03-25T18:29:55.938 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.934+0000 7f000bf8d640 1 -- 172.21.15.155:0/2989807869 >> 172.21.15.155:0/2989807869 conn(0x7f00040b7ea0 msgr2=0x7f00040b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:55.938 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.934+0000 7f000bf8d640 1 -- 172.21.15.155:0/2989807869 shutdown_connections 2024-03-25T18:29:55.938 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.934+0000 7f000bf8d640 1 -- 172.21.15.155:0/2989807869 wait complete. 2024-03-25T18:29:55.938 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.935+0000 7f000bf8d640 10 monclient: build_initial_monmap 2024-03-25T18:29:55.938 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.935+0000 7f000bf8d640 10 monclient: monmap: 2024-03-25T18:29:55.938 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:29:55.938 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:55.938 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:29:55.938 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:29:55.938 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:55.938 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:29:55.938 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:29:55.939 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:29:55.939 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:29:55.939 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:29:55.939 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.935+0000 7f000bf8d640 1 librados: starting msgr at 2024-03-25T18:29:55.939 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.935+0000 7f000bf8d640 1 librados: starting objecter 2024-03-25T18:29:55.939 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.935+0000 7f000bf8d640 1 Processor -- start 2024-03-25T18:29:55.939 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.936+0000 7f000bf8d640 1 -- start start 2024-03-25T18:29:55.939 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.936+0000 7f000bf8d640 1 librados: setting wanted keys 2024-03-25T18:29:55.939 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.936+0000 7f000bf8d640 1 librados: calling monclient init 2024-03-25T18:29:55.939 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.936+0000 7f000bf8d640 10 monclient: init 2024-03-25T18:29:55.939 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.936+0000 7f000bf8d640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:55.939 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.936+0000 7f000bf8d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:29:55.939 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.936+0000 7f000bf8d640 10 monclient(hunting): picked mon.noname-c con 0x7f00040a98f0 addr v1:172.21.15.155:6790/0 2024-03-25T18:29:55.939 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.936+0000 7f000bf8d640 10 monclient(hunting): picked mon.noname-b con 0x7f000413f710 addr v1:172.21.15.167:6789/0 2024-03-25T18:29:55.940 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.936+0000 7f000bf8d640 10 monclient(hunting): picked mon.noname-a con 0x7f00041430c0 addr v1:172.21.15.155:6789/0 2024-03-25T18:29:55.940 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.936+0000 7f000bf8d640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0004137080 con 0x7f00041430c0 2024-03-25T18:29:55.940 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.936+0000 7f000bf8d640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0004137360 con 0x7f000413f710 2024-03-25T18:29:55.940 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.936+0000 7f000bf8d640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0004135b10 con 0x7f00040a98f0 2024-03-25T18:29:55.940 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.936+0000 7f000bf8d640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:55.940 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.936+0000 7f000bf8d640 10 monclient(hunting): authenticate will time out at 7842.246401s 2024-03-25T18:29:55.946 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.945+0000 7f000a52a640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f00041430c0 0x7f0004130340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:35290/0 (socket says 172.21.15.155:35290) 2024-03-25T18:29:55.946 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.945+0000 7f000a52a640 1 -- 172.21.15.155:0/3437987046 learned_addr learned my addr 172.21.15.155:0/3437987046 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:29:55.946 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.945+0000 7efffa7fc640 10 client.?.objecter ms_handle_connect 0x7f00041430c0 2024-03-25T18:29:55.946 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.945+0000 7efffa7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:55.953 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.952+0000 7efffa7fc640 10 client.?.objecter ms_handle_connect 0x7f000413f710 2024-03-25T18:29:55.953 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.952+0000 7efffa7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:55.954 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.953+0000 7efffa7fc640 1 -- 172.21.15.155:0/3437987046 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3318820147 0 0) 0x7f0004137080 con 0x7f00041430c0 2024-03-25T18:29:55.954 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.953+0000 7efffa7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:29:55.954 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.953+0000 7efffa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:55.954 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.953+0000 7efffa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:55.954 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.953+0000 7efffa7fc640 10 monclient(hunting): my global_id is 4458 2024-03-25T18:29:55.954 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.953+0000 7efffa7fc640 1 -- 172.21.15.155:0/3437987046 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effe8004eb0 con 0x7f00041430c0 2024-03-25T18:29:55.962 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.961+0000 7efffa7fc640 10 client.?.objecter ms_handle_connect 0x7f00040a98f0 2024-03-25T18:29:55.962 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.961+0000 7efffa7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:55.966 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.965+0000 7efffa7fc640 1 -- 172.21.15.155:0/3437987046 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3262031578 0 0) 0x7f0004137360 con 0x7f000413f710 2024-03-25T18:29:55.967 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.965+0000 7efffa7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:29:55.967 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.965+0000 7efffa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:55.967 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.965+0000 7efffa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:55.967 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.965+0000 7efffa7fc640 10 monclient(hunting): my global_id is 4450 2024-03-25T18:29:55.967 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.965+0000 7efffa7fc640 1 -- 172.21.15.155:0/3437987046 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0004137080 con 0x7f000413f710 2024-03-25T18:29:55.971 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.970+0000 7efffa7fc640 1 -- 172.21.15.155:0/3437987046 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2401921143 0 0) 0x7effe8004eb0 con 0x7f00041430c0 2024-03-25T18:29:55.971 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.970+0000 7efffa7fc640 1 -- 172.21.15.155:0/3437987046 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0004137360 con 0x7f00041430c0 2024-03-25T18:29:55.971 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.970+0000 7efffa7fc640 1 -- 172.21.15.155:0/3437987046 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7efffc001c90 con 0x7f00041430c0 2024-03-25T18:29:55.971 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.970+0000 7efffa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:55.972 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.970+0000 7efffa7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:29:55.972 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.970+0000 7efffa7fc640 10 monclient(hunting): dump: 2024-03-25T18:29:55.972 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:29:55.972 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:29:55.972 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:55.972 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:55.972 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:55.972 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:29:55.972 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:29:55.973 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:29:55.973 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:29:55.973 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:29:55.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.970+0000 7efffa7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:55.974 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.971+0000 7efffa7fc640 1 -- 172.21.15.155:0/3437987046 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1959798366 0 0) 0x7f0004135b10 con 0x7f00040a98f0 2024-03-25T18:29:55.974 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.971+0000 7efffa7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:29:55.974 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.971+0000 7efffa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:55.974 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.971+0000 7efffa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:55.974 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.971+0000 7efffa7fc640 10 monclient(hunting): my global_id is 4463 2024-03-25T18:29:55.974 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.971+0000 7efffa7fc640 1 -- 172.21.15.155:0/3437987046 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effe8004eb0 con 0x7f00040a98f0 2024-03-25T18:29:55.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.984+0000 7efffa7fc640 1 -- 172.21.15.155:0/3437987046 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 459640021 0 0) 0x7f0004137360 con 0x7f00041430c0 2024-03-25T18:29:55.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.984+0000 7efffa7fc640 1 -- 172.21.15.155:0/3437987046 >> v1:172.21.15.155:6790/0 conn(0x7f00040a98f0 legacy=0x7f0004137a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:55.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.984+0000 7efffa7fc640 1 -- 172.21.15.155:0/3437987046 >> v1:172.21.15.167:6789/0 conn(0x7f000413f710 legacy=0x7f00040b6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:55.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.984+0000 7efffa7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:29:55.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.984+0000 7efffa7fc640 1 monclient: found mon.a 2024-03-25T18:29:55.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.984+0000 7efffa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:55.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.984+0000 7efffa7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:29:55.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.984+0000 7efffa7fc640 1 -- 172.21.15.155:0/3437987046 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0004135df0 con 0x7f00041430c0 2024-03-25T18:29:55.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.984+0000 7efffa7fc640 10 monclient: _finish_auth 0 2024-03-25T18:29:55.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.984+0000 7efffa7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:29:55.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.984+0000 7efffa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:55.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.984+0000 7f000bf8d640 5 monclient: authenticate success, global_id 4458 2024-03-25T18:29:55.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.984+0000 7f000bf8d640 10 monclient: _renew_subs 2024-03-25T18:29:55.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.984+0000 7f000bf8d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:29:55.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.984+0000 7f000bf8d640 1 -- 172.21.15.155:0/3437987046 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0004136340 con 0x7f00041430c0 2024-03-25T18:29:55.987 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.984+0000 7f000bf8d640 10 client.4458.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:55.987 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.984+0000 7f000bf8d640 10 monclient: _renew_subs 2024-03-25T18:29:55.987 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.984+0000 7f000bf8d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:29:55.987 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.984+0000 7f000bf8d640 1 -- 172.21.15.155:0/3437987046 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f00041352e0 con 0x7f00041430c0 2024-03-25T18:29:55.987 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.984+0000 7f000bf8d640 1 librados: init done 2024-03-25T18:29:55.987 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.985+0000 7f000bf8d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:55.987 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.985+0000 7f000bf8d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:55.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.985+0000 7f000bf8d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:29:55.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:55.985+0000 7f000bf8d640 1 -- 172.21.15.155:0/3437987046 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00040627b0 con 0x7f00041430c0 2024-03-25T18:29:56.328 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:56.327+0000 7efffa7fc640 1 -- 172.21.15.155:0/3437987046 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efffc003970 con 0x7f00041430c0 2024-03-25T18:29:56.328 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:56.327+0000 7efffa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:56.328 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:56.327+0000 7efffa7fc640 1 -- 172.21.15.155:0/3437987046 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7efffc003d80 con 0x7f00041430c0 2024-03-25T18:29:56.329 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:56.327+0000 7efffa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:56.329 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:56.327+0000 7efffa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:56.329 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:56.327+0000 7efffa7fc640 10 monclient: dump: 2024-03-25T18:29:56.329 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:29:56.329 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:29:56.329 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:56.329 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:29:56.329 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:56.329 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:29:56.329 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:29:56.329 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:29:56.329 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:29:56.329 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:29:56.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:56.327+0000 7efffbfff640 4 set_mon_vals no callback set 2024-03-25T18:29:56.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:56.328+0000 7efffa7fc640 1 -- 172.21.15.155:0/3437987046 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7efffc003fa0 con 0x7f00041430c0 2024-03-25T18:29:56.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:56.329+0000 7f0009d29640 10 monclient: get_auth_request con 0x7effe806de00 auth_method 0 2024-03-25T18:29:56.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:56.329+0000 7efffa7fc640 10 client.4458.objecter ms_handle_connect 0x7effe806de00 2024-03-25T18:29:56.334 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:56.333+0000 7efffa7fc640 1 -- 172.21.15.155:0/3437987046 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7efffc054ac0 con 0x7f00041430c0 2024-03-25T18:29:56.335 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:56.333+0000 7efffa7fc640 10 client.4458.objecter ms_dispatch 0x7f0004000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:56.335 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:56.333+0000 7efffa7fc640 3 client.4458.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:56.335 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:56.333+0000 7efffa7fc640 3 client.4458.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:56.335 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:56.333+0000 7efffa7fc640 20 client.4458.objecter dump_active .. 0 homeless 2024-03-25T18:29:56.335 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:56.333+0000 7efffa7fc640 1 -- 172.21.15.155:0/3437987046 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efffc0505d0 con 0x7f00041430c0 2024-03-25T18:29:56.335 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:56.333+0000 7efffa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:56.335 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:56.333+0000 7efffa7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:56.591 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:56.590+0000 7f000bf8d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:56.591 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:56.590+0000 7f000bf8d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:56.592 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:56.590+0000 7f000bf8d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:29:56.592 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:56.590+0000 7f000bf8d640 1 -- 172.21.15.155:0/3437987046 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f000414f110 con 0x7f00041430c0 2024-03-25T18:29:56.619 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:56.618+0000 7efffa7fc640 1 -- 172.21.15.155:0/3437987046 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7efffc052500 con 0x7f00041430c0 2024-03-25T18:29:56.619 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:56.618+0000 7efffa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:56.619 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:56.618+0000 7efffa7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:56.619 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:29:56.619 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:56.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:56.620+0000 7f000bf8d640 10 librados: watch_flush enter 2024-03-25T18:29:56.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:56.620+0000 7f000bf8d640 10 librados: watch_flush exit 2024-03-25T18:29:56.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:56.620+0000 7f000bf8d640 20 client.4458.objecter shutdown clearing up homeless session... 2024-03-25T18:29:56.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:56.620+0000 7f000bf8d640 10 client.4458.objecter successfully canceled tick 2024-03-25T18:29:56.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:56.620+0000 7f000bf8d640 1 -- 172.21.15.155:0/3437987046 >> v1:172.21.15.167:6800/2919070503 conn(0x7effe806de00 legacy=0x7effe80702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:56.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:56.620+0000 7f000bf8d640 10 monclient: shutdown 2024-03-25T18:29:56.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:56.620+0000 7f000bf8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:56.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:56.620+0000 7f000bf8d640 1 -- 172.21.15.155:0/3437987046 >> v1:172.21.15.155:6789/0 conn(0x7f00041430c0 legacy=0x7f0004130340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:56.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:56.620+0000 7f000bf8d640 1 -- 172.21.15.155:0/3437987046 shutdown_connections 2024-03-25T18:29:56.623 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:56.620+0000 7f000bf8d640 1 -- 172.21.15.155:0/3437987046 >> 172.21.15.155:0/3437987046 conn(0x7f00040b7ea0 msgr2=0x7f0004074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:56.623 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:56.620+0000 7f000bf8d640 1 -- 172.21.15.155:0/3437987046 shutdown_connections 2024-03-25T18:29:56.623 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:56.621+0000 7f000bf8d640 1 -- 172.21.15.155:0/3437987046 wait complete. 2024-03-25T18:29:56.623 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:29:56.621+0000 7f000bf8d640 1 librados: shutdown 2024-03-25T18:29:56.633 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:03.440 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:03.555 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.554+0000 7f9d6af6b640 10 monclient: get_monmap_and_config 2024-03-25T18:30:03.555 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.554+0000 7f9d6af6b640 10 monclient: build_initial_monmap 2024-03-25T18:30:03.555 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.554+0000 7f9d6af6b640 10 monclient: monmap: 2024-03-25T18:30:03.556 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:30:03.556 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:30:03.556 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:30:03.555291+0000 2024-03-25T18:30:03.556 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:30:03.555291+0000 2024-03-25T18:30:03.556 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:03.556 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:30:03.556 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:30:03.556 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:30:03.556 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:30:03.556 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:30:03.557 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.555+0000 7f9d6af6b640 1 Processor -- start 2024-03-25T18:30:03.557 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.555+0000 7f9d6af6b640 1 -- start start 2024-03-25T18:30:03.558 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.555+0000 7f9d6af6b640 10 monclient: init 2024-03-25T18:30:03.558 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.556+0000 7f9d6af6b640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:03.558 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.556+0000 7f9d6af6b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:30:03.558 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.556+0000 7f9d6af6b640 10 monclient(hunting): picked mon.noname-a con 0x7f9d640a98f0 addr v1:172.21.15.155:6789/0 2024-03-25T18:30:03.558 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.556+0000 7f9d6af6b640 10 monclient(hunting): picked mon.noname-c con 0x7f9d6413f710 addr v1:172.21.15.167:6789/0 2024-03-25T18:30:03.558 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.556+0000 7f9d6af6b640 10 monclient(hunting): picked mon.noname-b con 0x7f9d641430c0 addr v1:172.21.15.155:6790/0 2024-03-25T18:30:03.558 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.556+0000 7f9d6af6b640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d64142520 con 0x7f9d640a98f0 2024-03-25T18:30:03.558 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.556+0000 7f9d6af6b640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d640b28a0 con 0x7f9d6413f710 2024-03-25T18:30:03.558 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.556+0000 7f9d6af6b640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d640b2a30 con 0x7f9d641430c0 2024-03-25T18:30:03.558 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.556+0000 7f9d6af6b640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:03.558 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.556+0000 7f9d6af6b640 10 monclient(hunting): authenticate will time out at 7849.865885s 2024-03-25T18:30:03.571 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.570+0000 7f9d63fff640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f9d6413f710 0x7f9d641429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:33820/0 (socket says 172.21.15.155:33820) 2024-03-25T18:30:03.572 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.570+0000 7f9d63fff640 1 -- 172.21.15.155:0/996436815 learned_addr learned my addr 172.21.15.155:0/996436815 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:30:03.584 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.583+0000 7f9d637fe640 1 -- 172.21.15.155:0/996436815 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 853741741 0 0) 0x7f9d640b2a30 con 0x7f9d641430c0 2024-03-25T18:30:03.584 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.583+0000 7f9d637fe640 10 monclient(hunting): _negotiate 2024-03-25T18:30:03.584 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.583+0000 7f9d637fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:03.584 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.583+0000 7f9d637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:03.584 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.583+0000 7f9d637fe640 10 monclient(hunting): my global_id is 4466 2024-03-25T18:30:03.584 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.583+0000 7f9d637fe640 1 -- 172.21.15.155:0/996436815 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d54003520 con 0x7f9d641430c0 2024-03-25T18:30:03.588 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.587+0000 7f9d637fe640 1 -- 172.21.15.155:0/996436815 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 295599991 0 0) 0x7f9d64142520 con 0x7f9d640a98f0 2024-03-25T18:30:03.588 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.587+0000 7f9d637fe640 10 monclient(hunting): _negotiate 2024-03-25T18:30:03.588 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.587+0000 7f9d637fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:03.588 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.587+0000 7f9d637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:03.588 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.587+0000 7f9d637fe640 10 monclient(hunting): my global_id is 4461 2024-03-25T18:30:03.588 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.587+0000 7f9d637fe640 1 -- 172.21.15.155:0/996436815 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d640b2a30 con 0x7f9d640a98f0 2024-03-25T18:30:03.591 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.589+0000 7f9d637fe640 1 -- 172.21.15.155:0/996436815 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 463020957 0 0) 0x7f9d640b28a0 con 0x7f9d6413f710 2024-03-25T18:30:03.591 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.589+0000 7f9d637fe640 10 monclient(hunting): _negotiate 2024-03-25T18:30:03.591 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.589+0000 7f9d637fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:03.591 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.589+0000 7f9d637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:03.591 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.589+0000 7f9d637fe640 10 monclient(hunting): my global_id is 4453 2024-03-25T18:30:03.592 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.589+0000 7f9d637fe640 1 -- 172.21.15.155:0/996436815 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d64142520 con 0x7f9d6413f710 2024-03-25T18:30:03.610 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.608+0000 7f9d637fe640 1 -- 172.21.15.155:0/996436815 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1860372776 0 0) 0x7f9d64142520 con 0x7f9d6413f710 2024-03-25T18:30:03.610 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.608+0000 7f9d637fe640 1 -- 172.21.15.155:0/996436815 >> v1:172.21.15.155:6790/0 conn(0x7f9d641430c0 legacy=0x7f9d6414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:03.610 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.608+0000 7f9d637fe640 1 -- 172.21.15.155:0/996436815 >> v1:172.21.15.155:6789/0 conn(0x7f9d640a98f0 legacy=0x7f9d640a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:03.610 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.608+0000 7f9d637fe640 10 monclient: _finish_hunting 0 2024-03-25T18:30:03.610 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.608+0000 7f9d637fe640 1 monclient: found mon.noname-c 2024-03-25T18:30:03.610 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.608+0000 7f9d637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:03.610 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.608+0000 7f9d637fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.167:6789/0 2024-03-25T18:30:03.610 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.608+0000 7f9d637fe640 1 -- 172.21.15.155:0/996436815 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d640b44c0 con 0x7f9d6413f710 2024-03-25T18:30:03.610 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.608+0000 7f9d637fe640 10 monclient: _finish_auth 0 2024-03-25T18:30:03.610 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.608+0000 7f9d637fe640 10 monclient: _check_auth_tickets 2024-03-25T18:30:03.610 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.608+0000 7f9d637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:03.610 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.608+0000 7f9d637fe640 1 -- 172.21.15.155:0/996436815 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f9d500015b0 con 0x7f9d6413f710 2024-03-25T18:30:03.611 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.608+0000 7f9d637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:03.611 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.608+0000 7f9d637fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:30:03.611 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.608+0000 7f9d637fe640 10 monclient: dump: 2024-03-25T18:30:03.611 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:30:03.611 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:30:03.611 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:03.611 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:03.611 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:03.611 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:30:03.611 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:30:03.611 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:30:03.611 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:30:03.611 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:30:03.611 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.608+0000 7f9d6af6b640 5 monclient: authenticate success, global_id 4453 2024-03-25T18:30:03.612 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.608+0000 7f9d6af6b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:03.632 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.631+0000 7f9d637fe640 1 -- 172.21.15.155:0/996436815 <== mon.1 v1:172.21.15.167:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9d50001710 con 0x7f9d6413f710 2024-03-25T18:30:03.632 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.631+0000 7f9d637fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:03.632 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.631+0000 7f9d637fe640 1 -- 172.21.15.155:0/996436815 <== mon.1 v1:172.21.15.167:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f9d50001b60 con 0x7f9d6413f710 2024-03-25T18:30:03.632 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.631+0000 7f9d6af6b640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:03.632 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.631+0000 7f9d6af6b640 4 set_mon_vals no callback set 2024-03-25T18:30:03.633 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.631+0000 7f9d637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:03.633 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.631+0000 7f9d637fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:03.633 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.631+0000 7f9d637fe640 10 monclient: dump: 2024-03-25T18:30:03.633 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:30:03.633 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:30:03.633 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:03.633 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:03.633 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:03.633 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:30:03.633 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:30:03.633 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:30:03.633 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:30:03.633 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:30:03.635 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.632+0000 7f9d6af6b640 10 monclient: shutdown 2024-03-25T18:30:03.635 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.632+0000 7f9d6af6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:03.635 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.632+0000 7f9d6af6b640 1 -- 172.21.15.155:0/996436815 >> v1:172.21.15.167:6789/0 conn(0x7f9d6413f710 legacy=0x7f9d641429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:03.635 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.632+0000 7f9d6af6b640 1 -- 172.21.15.155:0/996436815 shutdown_connections 2024-03-25T18:30:03.635 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.632+0000 7f9d6af6b640 1 -- 172.21.15.155:0/996436815 >> 172.21.15.155:0/996436815 conn(0x7f9d640b7ea0 msgr2=0x7f9d640b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:03.635 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.632+0000 7f9d6af6b640 1 -- 172.21.15.155:0/996436815 shutdown_connections 2024-03-25T18:30:03.635 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.632+0000 7f9d6af6b640 1 -- 172.21.15.155:0/996436815 wait complete. 2024-03-25T18:30:03.635 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.633+0000 7f9d6af6b640 10 monclient: build_initial_monmap 2024-03-25T18:30:03.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.633+0000 7f9d6af6b640 10 monclient: monmap: 2024-03-25T18:30:03.636 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:30:03.636 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:03.636 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:30:03.636 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:30:03.636 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:03.636 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:30:03.636 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:30:03.636 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:30:03.636 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:30:03.636 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:30:03.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.634+0000 7f9d6af6b640 1 librados: starting msgr at 2024-03-25T18:30:03.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.634+0000 7f9d6af6b640 1 librados: starting objecter 2024-03-25T18:30:03.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.634+0000 7f9d6af6b640 1 Processor -- start 2024-03-25T18:30:03.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.634+0000 7f9d6af6b640 1 -- start start 2024-03-25T18:30:03.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.634+0000 7f9d6af6b640 1 librados: setting wanted keys 2024-03-25T18:30:03.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.634+0000 7f9d6af6b640 1 librados: calling monclient init 2024-03-25T18:30:03.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.634+0000 7f9d6af6b640 10 monclient: init 2024-03-25T18:30:03.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.635+0000 7f9d6af6b640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:03.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.635+0000 7f9d6af6b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:30:03.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.635+0000 7f9d6af6b640 10 monclient(hunting): picked mon.noname-c con 0x7f9d640a98f0 addr v1:172.21.15.155:6790/0 2024-03-25T18:30:03.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.635+0000 7f9d6af6b640 10 monclient(hunting): picked mon.noname-a con 0x7f9d6413f710 addr v1:172.21.15.155:6789/0 2024-03-25T18:30:03.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.635+0000 7f9d6af6b640 10 monclient(hunting): picked mon.noname-b con 0x7f9d641430c0 addr v1:172.21.15.167:6789/0 2024-03-25T18:30:03.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.635+0000 7f9d6af6b640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d641339d0 con 0x7f9d6413f710 2024-03-25T18:30:03.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.635+0000 7f9d6af6b640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d64138180 con 0x7f9d641430c0 2024-03-25T18:30:03.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.635+0000 7f9d6af6b640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d64138310 con 0x7f9d640a98f0 2024-03-25T18:30:03.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.635+0000 7f9d6af6b640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:03.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.635+0000 7f9d6af6b640 10 monclient(hunting): authenticate will time out at 7849.945164s 2024-03-25T18:30:03.641 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.641+0000 7f9d68d07640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f9d640a98f0 0x7f9d641d7500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:51324/0 (socket says 172.21.15.155:51324) 2024-03-25T18:30:03.642 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.641+0000 7f9d68d07640 1 -- 172.21.15.155:0/1813807257 learned_addr learned my addr 172.21.15.155:0/1813807257 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:30:03.643 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.642+0000 7f9d617fa640 10 client.?.objecter ms_handle_connect 0x7f9d640a98f0 2024-03-25T18:30:03.643 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.642+0000 7f9d617fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:03.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.646+0000 7f9d617fa640 1 -- 172.21.15.155:0/1813807257 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3656725177 0 0) 0x7f9d64138310 con 0x7f9d640a98f0 2024-03-25T18:30:03.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.646+0000 7f9d617fa640 10 monclient(hunting): _negotiate 2024-03-25T18:30:03.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.646+0000 7f9d617fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:03.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.646+0000 7f9d617fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:03.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.646+0000 7f9d617fa640 10 monclient(hunting): my global_id is 4469 2024-03-25T18:30:03.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.646+0000 7f9d617fa640 1 -- 172.21.15.155:0/1813807257 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d48004eb0 con 0x7f9d640a98f0 2024-03-25T18:30:03.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.647+0000 7f9d617fa640 10 client.?.objecter ms_handle_connect 0x7f9d6413f710 2024-03-25T18:30:03.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.647+0000 7f9d617fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:03.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.648+0000 7f9d617fa640 10 client.?.objecter ms_handle_connect 0x7f9d641430c0 2024-03-25T18:30:03.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.648+0000 7f9d617fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:03.654 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.653+0000 7f9d617fa640 1 -- 172.21.15.155:0/1813807257 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1872759329 0 0) 0x7f9d641339d0 con 0x7f9d6413f710 2024-03-25T18:30:03.654 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.653+0000 7f9d617fa640 10 monclient(hunting): _negotiate 2024-03-25T18:30:03.654 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.653+0000 7f9d617fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:03.654 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.653+0000 7f9d617fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:03.654 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.653+0000 7f9d617fa640 10 monclient(hunting): my global_id is 4464 2024-03-25T18:30:03.654 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.653+0000 7f9d617fa640 1 -- 172.21.15.155:0/1813807257 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d64138310 con 0x7f9d6413f710 2024-03-25T18:30:03.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.659+0000 7f9d617fa640 1 -- 172.21.15.155:0/1813807257 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2708437703 0 0) 0x7f9d64138180 con 0x7f9d641430c0 2024-03-25T18:30:03.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.659+0000 7f9d617fa640 10 monclient(hunting): _negotiate 2024-03-25T18:30:03.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.659+0000 7f9d617fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:03.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.659+0000 7f9d617fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:03.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.659+0000 7f9d617fa640 10 monclient(hunting): my global_id is 4456 2024-03-25T18:30:03.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.659+0000 7f9d617fa640 1 -- 172.21.15.155:0/1813807257 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d641339d0 con 0x7f9d641430c0 2024-03-25T18:30:03.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.660+0000 7f9d617fa640 1 -- 172.21.15.155:0/1813807257 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3427277161 0 0) 0x7f9d48004eb0 con 0x7f9d640a98f0 2024-03-25T18:30:03.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.660+0000 7f9d617fa640 1 -- 172.21.15.155:0/1813807257 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d64138180 con 0x7f9d640a98f0 2024-03-25T18:30:03.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.660+0000 7f9d617fa640 1 -- 172.21.15.155:0/1813807257 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f9d58001a10 con 0x7f9d640a98f0 2024-03-25T18:30:03.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.660+0000 7f9d617fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:03.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.660+0000 7f9d617fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:30:03.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.660+0000 7f9d617fa640 10 monclient(hunting): dump: 2024-03-25T18:30:03.662 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:30:03.662 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:30:03.662 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:03.662 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:03.662 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:03.662 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:30:03.662 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:30:03.663 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:30:03.663 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:30:03.663 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:30:03.663 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.660+0000 7f9d617fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:03.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.664+0000 7f9d617fa640 1 -- 172.21.15.155:0/1813807257 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2029793164 0 0) 0x7f9d64138310 con 0x7f9d6413f710 2024-03-25T18:30:03.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.664+0000 7f9d617fa640 1 -- 172.21.15.155:0/1813807257 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d48004eb0 con 0x7f9d6413f710 2024-03-25T18:30:03.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.664+0000 7f9d617fa640 1 -- 172.21.15.155:0/1813807257 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f9d50001a20 con 0x7f9d6413f710 2024-03-25T18:30:03.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.664+0000 7f9d617fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:03.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.664+0000 7f9d617fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:30:03.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.664+0000 7f9d617fa640 10 monclient(hunting): dump: 2024-03-25T18:30:03.665 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:30:03.665 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:30:03.665 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:03.665 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:03.665 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:03.666 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:30:03.666 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:30:03.666 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:30:03.666 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:30:03.666 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:30:03.675 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.673+0000 7f9d617fa640 1 -- 172.21.15.155:0/1813807257 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1283034068 0 0) 0x7f9d64138180 con 0x7f9d640a98f0 2024-03-25T18:30:03.675 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.674+0000 7f9d617fa640 1 -- 172.21.15.155:0/1813807257 >> v1:172.21.15.167:6789/0 conn(0x7f9d641430c0 legacy=0x7f9d641332c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:03.675 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.674+0000 7f9d617fa640 1 -- 172.21.15.155:0/1813807257 >> v1:172.21.15.155:6789/0 conn(0x7f9d6413f710 legacy=0x7f9d64132bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:03.676 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.674+0000 7f9d617fa640 10 monclient: _finish_hunting 0 2024-03-25T18:30:03.676 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.674+0000 7f9d617fa640 1 monclient: found mon.c 2024-03-25T18:30:03.676 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.674+0000 7f9d617fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:03.676 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.674+0000 7f9d617fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:30:03.676 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.674+0000 7f9d617fa640 1 -- 172.21.15.155:0/1813807257 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d64139760 con 0x7f9d640a98f0 2024-03-25T18:30:03.676 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.674+0000 7f9d617fa640 10 monclient: _finish_auth 0 2024-03-25T18:30:03.676 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.674+0000 7f9d617fa640 10 monclient: _check_auth_tickets 2024-03-25T18:30:03.676 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.674+0000 7f9d617fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:03.676 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.674+0000 7f9d6af6b640 5 monclient: authenticate success, global_id 4469 2024-03-25T18:30:03.676 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.674+0000 7f9d6af6b640 10 monclient: _renew_subs 2024-03-25T18:30:03.676 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.674+0000 7f9d6af6b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:30:03.676 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.674+0000 7f9d6af6b640 1 -- 172.21.15.155:0/1813807257 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d64137810 con 0x7f9d640a98f0 2024-03-25T18:30:03.676 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.674+0000 7f9d6af6b640 10 client.4469.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:03.676 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.674+0000 7f9d6af6b640 10 monclient: _renew_subs 2024-03-25T18:30:03.677 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.674+0000 7f9d6af6b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:30:03.677 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.674+0000 7f9d6af6b640 1 -- 172.21.15.155:0/1813807257 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d641d7df0 con 0x7f9d640a98f0 2024-03-25T18:30:03.677 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.674+0000 7f9d6af6b640 1 librados: init done 2024-03-25T18:30:03.677 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.675+0000 7f9d6af6b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:03.677 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.675+0000 7f9d6af6b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:03.677 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.675+0000 7f9d6af6b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:30:03.678 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.675+0000 7f9d6af6b640 1 -- 172.21.15.155:0/1813807257 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d640627b0 con 0x7f9d640a98f0 2024-03-25T18:30:03.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.992+0000 7f9d617fa640 1 -- 172.21.15.155:0/1813807257 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9d58001c50 con 0x7f9d640a98f0 2024-03-25T18:30:03.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.992+0000 7f9d617fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:03.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.992+0000 7f9d617fa640 1 -- 172.21.15.155:0/1813807257 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f9d58003a60 con 0x7f9d640a98f0 2024-03-25T18:30:03.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.992+0000 7f9d617fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:03.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.992+0000 7f9d617fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:30:03.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.992+0000 7f9d617fa640 10 monclient: dump: 2024-03-25T18:30:03.994 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:30:03.994 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:30:03.994 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:03.994 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:03.994 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:03.994 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:30:03.994 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:30:03.994 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:30:03.995 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:30:03.995 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:30:03.995 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.992+0000 7f9d62ffd640 4 set_mon_vals no callback set 2024-03-25T18:30:03.995 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.993+0000 7f9d617fa640 1 -- 172.21.15.155:0/1813807257 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7f9d58003c80 con 0x7f9d640a98f0 2024-03-25T18:30:03.995 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.994+0000 7f9d63fff640 10 monclient: get_auth_request con 0x7f9d4806df70 auth_method 0 2024-03-25T18:30:03.995 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.994+0000 7f9d617fa640 10 client.4469.objecter ms_handle_connect 0x7f9d4806df70 2024-03-25T18:30:04.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.999+0000 7f9d617fa640 1 -- 172.21.15.155:0/1813807257 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7f9d58054880 con 0x7f9d640a98f0 2024-03-25T18:30:04.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.999+0000 7f9d617fa640 10 client.4469.objecter ms_dispatch 0x7f9d64000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:30:04.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.999+0000 7f9d617fa640 3 client.4469.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:30:04.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.999+0000 7f9d617fa640 3 client.4469.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:30:04.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.999+0000 7f9d617fa640 20 client.4469.objecter dump_active .. 0 homeless 2024-03-25T18:30:04.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.999+0000 7f9d617fa640 1 -- 172.21.15.155:0/1813807257 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9d58050390 con 0x7f9d640a98f0 2024-03-25T18:30:04.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.999+0000 7f9d617fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:04.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:03.999+0000 7f9d617fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:04.235 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:04.233+0000 7f9d6af6b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:04.235 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:04.233+0000 7f9d6af6b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:04.235 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:04.233+0000 7f9d6af6b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:30:04.235 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:04.233+0000 7f9d6af6b640 1 -- 172.21.15.155:0/1813807257 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9d641379a0 con 0x7f9d640a98f0 2024-03-25T18:30:04.339 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:04.337+0000 7f9d617fa640 1 -- 172.21.15.155:0/1813807257 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7f9d580522c0 con 0x7f9d640a98f0 2024-03-25T18:30:04.339 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:04.338+0000 7f9d617fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:04.339 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:04.338+0000 7f9d617fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:04.339 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:30:04.339 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:04.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:04.340+0000 7f9d6af6b640 10 librados: watch_flush enter 2024-03-25T18:30:04.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:04.340+0000 7f9d6af6b640 10 librados: watch_flush exit 2024-03-25T18:30:04.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:04.340+0000 7f9d6af6b640 20 client.4469.objecter shutdown clearing up homeless session... 2024-03-25T18:30:04.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:04.340+0000 7f9d6af6b640 10 client.4469.objecter successfully canceled tick 2024-03-25T18:30:04.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:04.340+0000 7f9d6af6b640 1 -- 172.21.15.155:0/1813807257 >> v1:172.21.15.167:6800/2919070503 conn(0x7f9d4806df70 legacy=0x7f9d48070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:04.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:04.340+0000 7f9d6af6b640 10 monclient: shutdown 2024-03-25T18:30:04.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:04.340+0000 7f9d6af6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:04.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:04.340+0000 7f9d6af6b640 1 -- 172.21.15.155:0/1813807257 >> v1:172.21.15.155:6790/0 conn(0x7f9d640a98f0 legacy=0x7f9d641d7500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:04.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:04.340+0000 7f9d6af6b640 1 -- 172.21.15.155:0/1813807257 shutdown_connections 2024-03-25T18:30:04.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:04.340+0000 7f9d6af6b640 1 -- 172.21.15.155:0/1813807257 >> 172.21.15.155:0/1813807257 conn(0x7f9d640b7ea0 msgr2=0x7f9d64074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:04.343 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:04.341+0000 7f9d6af6b640 1 -- 172.21.15.155:0/1813807257 shutdown_connections 2024-03-25T18:30:04.343 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:04.341+0000 7f9d6af6b640 1 -- 172.21.15.155:0/1813807257 wait complete. 2024-03-25T18:30:04.343 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:04.341+0000 7f9d6af6b640 1 librados: shutdown 2024-03-25T18:30:04.354 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:06.404 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:05:40.008 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:06.415 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:05:40.019 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:06.441 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:05:40.046 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:06.465 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:05:40.044 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:06.468 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:05:40.073 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:06.473 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:05:40.052 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:06.479 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:05:40.084 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:06.506 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:05:40.111 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:06.509 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:05:40.087 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:11.163 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:11.272 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.271+0000 7fb8a509d640 10 monclient: get_monmap_and_config 2024-03-25T18:30:11.272 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.271+0000 7fb8a509d640 10 monclient: build_initial_monmap 2024-03-25T18:30:11.272 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.271+0000 7fb8a509d640 10 monclient: monmap: 2024-03-25T18:30:11.272 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:30:11.272 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:30:11.273 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:30:11.272274+0000 2024-03-25T18:30:11.273 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:30:11.272274+0000 2024-03-25T18:30:11.273 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:11.273 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:30:11.273 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:30:11.273 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:30:11.273 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:30:11.273 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:30:11.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.272+0000 7fb8a509d640 1 Processor -- start 2024-03-25T18:30:11.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.272+0000 7fb8a509d640 1 -- start start 2024-03-25T18:30:11.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.272+0000 7fb8a509d640 10 monclient: init 2024-03-25T18:30:11.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.273+0000 7fb8a509d640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:11.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.273+0000 7fb8a509d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:30:11.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.273+0000 7fb8a509d640 10 monclient(hunting): picked mon.noname-b con 0x7fb8a00b4d60 addr v1:172.21.15.155:6790/0 2024-03-25T18:30:11.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.273+0000 7fb8a509d640 10 monclient(hunting): picked mon.noname-c con 0x7fb8a00b1730 addr v1:172.21.15.167:6789/0 2024-03-25T18:30:11.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.273+0000 7fb8a509d640 10 monclient(hunting): picked mon.noname-a con 0x7fb8a014be90 addr v1:172.21.15.155:6789/0 2024-03-25T18:30:11.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.273+0000 7fb8a509d640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8a00af940 con 0x7fb8a014be90 2024-03-25T18:30:11.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.273+0000 7fb8a509d640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8a00b0a70 con 0x7fb8a00b1730 2024-03-25T18:30:11.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.273+0000 7fb8a509d640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8a00b0d50 con 0x7fb8a00b4d60 2024-03-25T18:30:11.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.273+0000 7fb8a509d640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:11.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.273+0000 7fb8a509d640 10 monclient(hunting): authenticate will time out at 7857.582884s 2024-03-25T18:30:11.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.273+0000 7fb897fff640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7fb8a00b1730 0x7fb8a00b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:41308/0 (socket says 172.21.15.155:41308) 2024-03-25T18:30:11.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.273+0000 7fb897fff640 1 -- 172.21.15.155:0/55844818 learned_addr learned my addr 172.21.15.155:0/55844818 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:30:11.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.339+0000 7fb89e59c640 1 -- 172.21.15.155:0/55844818 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2943225027 0 0) 0x7fb8a00af940 con 0x7fb8a014be90 2024-03-25T18:30:11.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.339+0000 7fb89e59c640 10 monclient(hunting): _negotiate 2024-03-25T18:30:11.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.339+0000 7fb89e59c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:11.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.339+0000 7fb89e59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:11.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.339+0000 7fb89e59c640 10 monclient(hunting): my global_id is 4467 2024-03-25T18:30:11.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.339+0000 7fb89e59c640 1 -- 172.21.15.155:0/55844818 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb88c003520 con 0x7fb8a014be90 2024-03-25T18:30:11.348 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.347+0000 7fb89e59c640 1 -- 172.21.15.155:0/55844818 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3461147731 0 0) 0x7fb88c003520 con 0x7fb8a014be90 2024-03-25T18:30:11.348 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.347+0000 7fb89e59c640 1 -- 172.21.15.155:0/55844818 >> v1:172.21.15.155:6790/0 conn(0x7fb8a00b4d60 legacy=0x7fb8a013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:11.348 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.347+0000 7fb89e59c640 1 -- 172.21.15.155:0/55844818 >> v1:172.21.15.167:6789/0 conn(0x7fb8a00b1730 legacy=0x7fb8a00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:11.348 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.347+0000 7fb89e59c640 10 monclient: _finish_hunting 0 2024-03-25T18:30:11.349 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.347+0000 7fb89e59c640 1 monclient: found mon.noname-a 2024-03-25T18:30:11.349 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.347+0000 7fb89e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:11.349 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.347+0000 7fb89e59c640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-25T18:30:11.349 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.347+0000 7fb89e59c640 1 -- 172.21.15.155:0/55844818 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8a00b1030 con 0x7fb8a014be90 2024-03-25T18:30:11.349 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.347+0000 7fb89e59c640 10 monclient: _finish_auth 0 2024-03-25T18:30:11.349 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.347+0000 7fb89e59c640 10 monclient: _check_auth_tickets 2024-03-25T18:30:11.349 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.347+0000 7fb89e59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:11.349 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.347+0000 7fb89e59c640 1 -- 172.21.15.155:0/55844818 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fb898001a10 con 0x7fb8a014be90 2024-03-25T18:30:11.349 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.347+0000 7fb8a509d640 5 monclient: authenticate success, global_id 4467 2024-03-25T18:30:11.349 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.347+0000 7fb8a509d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:11.349 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.347+0000 7fb89e59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:11.349 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.347+0000 7fb89e59c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:30:11.349 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.347+0000 7fb89e59c640 10 monclient: dump: 2024-03-25T18:30:11.349 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:30:11.349 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:30:11.350 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:11.350 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:11.350 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:11.350 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:30:11.350 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:30:11.350 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:30:11.350 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:30:11.350 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:30:11.350 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.347+0000 7fb8a509d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:11.357 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.356+0000 7fb89e59c640 1 -- 172.21.15.155:0/55844818 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb898001b70 con 0x7fb8a014be90 2024-03-25T18:30:11.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.356+0000 7fb89e59c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:11.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.356+0000 7fb89e59c640 1 -- 172.21.15.155:0/55844818 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fb898001f80 con 0x7fb8a014be90 2024-03-25T18:30:11.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.356+0000 7fb89e59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:11.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.356+0000 7fb89e59c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:30:11.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.356+0000 7fb89e59c640 10 monclient: dump: 2024-03-25T18:30:11.358 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:30:11.358 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:30:11.358 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:11.358 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:11.358 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:11.358 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:30:11.358 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:30:11.359 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:30:11.359 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:30:11.359 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:30:11.359 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.356+0000 7fb8a509d640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:11.359 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.356+0000 7fb8a509d640 4 set_mon_vals no callback set 2024-03-25T18:30:11.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.357+0000 7fb8a509d640 10 monclient: shutdown 2024-03-25T18:30:11.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.357+0000 7fb8a509d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:11.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.357+0000 7fb8a509d640 1 -- 172.21.15.155:0/55844818 >> v1:172.21.15.155:6789/0 conn(0x7fb8a014be90 legacy=0x7fb8a014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:11.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.357+0000 7fb8a509d640 1 -- 172.21.15.155:0/55844818 shutdown_connections 2024-03-25T18:30:11.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.357+0000 7fb8a509d640 1 -- 172.21.15.155:0/55844818 >> 172.21.15.155:0/55844818 conn(0x7fb8a00617a0 msgr2=0x7fb8a0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:11.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.357+0000 7fb8a509d640 1 -- 172.21.15.155:0/55844818 shutdown_connections 2024-03-25T18:30:11.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.358+0000 7fb8a509d640 1 -- 172.21.15.155:0/55844818 wait complete. 2024-03-25T18:30:11.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.358+0000 7fb8a509d640 10 monclient: build_initial_monmap 2024-03-25T18:30:11.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.358+0000 7fb8a509d640 10 monclient: monmap: 2024-03-25T18:30:11.361 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:30:11.361 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:11.361 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:30:11.361 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:30:11.361 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:11.361 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:30:11.361 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:30:11.361 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:30:11.361 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:30:11.361 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:30:11.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.359+0000 7fb8a509d640 1 librados: starting msgr at 2024-03-25T18:30:11.362 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.359+0000 7fb8a509d640 1 librados: starting objecter 2024-03-25T18:30:11.362 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.359+0000 7fb8a509d640 1 Processor -- start 2024-03-25T18:30:11.362 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.359+0000 7fb8a509d640 1 -- start start 2024-03-25T18:30:11.362 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.359+0000 7fb8a509d640 1 librados: setting wanted keys 2024-03-25T18:30:11.362 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.359+0000 7fb8a509d640 1 librados: calling monclient init 2024-03-25T18:30:11.362 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.359+0000 7fb8a509d640 10 monclient: init 2024-03-25T18:30:11.363 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.360+0000 7fb8a509d640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:11.363 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.360+0000 7fb8a509d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:30:11.363 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.360+0000 7fb8a509d640 10 monclient(hunting): picked mon.noname-b con 0x7fb8a00b1730 addr v1:172.21.15.167:6789/0 2024-03-25T18:30:11.363 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.360+0000 7fb8a509d640 10 monclient(hunting): picked mon.noname-c con 0x7fb8a00b4d60 addr v1:172.21.15.155:6790/0 2024-03-25T18:30:11.363 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.360+0000 7fb8a509d640 10 monclient(hunting): picked mon.noname-a con 0x7fb8a014be90 addr v1:172.21.15.155:6789/0 2024-03-25T18:30:11.363 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.360+0000 7fb8a509d640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8a0137400 con 0x7fb8a014be90 2024-03-25T18:30:11.363 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.360+0000 7fb8a509d640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8a0137590 con 0x7fb8a00b1730 2024-03-25T18:30:11.363 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.360+0000 7fb8a509d640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8a0140380 con 0x7fb8a00b4d60 2024-03-25T18:30:11.363 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.360+0000 7fb8a509d640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:11.363 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.360+0000 7fb8a509d640 10 monclient(hunting): authenticate will time out at 7857.669974s 2024-03-25T18:30:11.369 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.368+0000 7fb897fff640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fb8a00b4d60 0x7fb8a01d7690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:56714/0 (socket says 172.21.15.155:56714) 2024-03-25T18:30:11.369 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.368+0000 7fb897fff640 1 -- 172.21.15.155:0/4074560518 learned_addr learned my addr 172.21.15.155:0/4074560518 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:30:11.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.369+0000 7fb8977fe640 10 client.?.objecter ms_handle_connect 0x7fb8a00b4d60 2024-03-25T18:30:11.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.369+0000 7fb8977fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:11.381 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.380+0000 7fb8977fe640 10 client.?.objecter ms_handle_connect 0x7fb8a00b1730 2024-03-25T18:30:11.381 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.380+0000 7fb8977fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:11.396 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.395+0000 7fb8977fe640 10 client.?.objecter ms_handle_connect 0x7fb8a014be90 2024-03-25T18:30:11.396 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.395+0000 7fb8977fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:11.464 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.463+0000 7fb8977fe640 1 -- 172.21.15.155:0/4074560518 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3580492224 0 0) 0x7fb8a0140380 con 0x7fb8a00b4d60 2024-03-25T18:30:11.465 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.463+0000 7fb8977fe640 10 monclient(hunting): _negotiate 2024-03-25T18:30:11.465 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.463+0000 7fb8977fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:11.465 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.463+0000 7fb8977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:11.465 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.463+0000 7fb8977fe640 10 monclient(hunting): my global_id is 4475 2024-03-25T18:30:11.465 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.463+0000 7fb8977fe640 1 -- 172.21.15.155:0/4074560518 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb880004eb0 con 0x7fb8a00b4d60 2024-03-25T18:30:11.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.463+0000 7fb8977fe640 1 -- 172.21.15.155:0/4074560518 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2576752955 0 0) 0x7fb8a0137400 con 0x7fb8a014be90 2024-03-25T18:30:11.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.463+0000 7fb8977fe640 10 monclient(hunting): _negotiate 2024-03-25T18:30:11.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.463+0000 7fb8977fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:11.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.463+0000 7fb8977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:11.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.463+0000 7fb8977fe640 10 monclient(hunting): my global_id is 4470 2024-03-25T18:30:11.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.463+0000 7fb8977fe640 1 -- 172.21.15.155:0/4074560518 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8a0140380 con 0x7fb8a014be90 2024-03-25T18:30:11.479 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.477+0000 7fb8977fe640 1 -- 172.21.15.155:0/4074560518 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 690270957 0 0) 0x7fb880004eb0 con 0x7fb8a00b4d60 2024-03-25T18:30:11.479 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.477+0000 7fb8977fe640 1 -- 172.21.15.155:0/4074560518 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8a0137400 con 0x7fb8a00b4d60 2024-03-25T18:30:11.479 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.478+0000 7fb8977fe640 1 -- 172.21.15.155:0/4074560518 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fb8880018b0 con 0x7fb8a00b4d60 2024-03-25T18:30:11.479 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.478+0000 7fb8977fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:11.479 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.478+0000 7fb8977fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:30:11.479 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.478+0000 7fb8977fe640 10 monclient(hunting): dump: 2024-03-25T18:30:11.479 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:30:11.479 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:30:11.479 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:11.479 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:11.479 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:11.480 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:30:11.480 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:30:11.480 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:30:11.480 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:30:11.480 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:30:11.480 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.478+0000 7fb8977fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:11.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.487+0000 7fb8977fe640 1 -- 172.21.15.155:0/4074560518 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 692183622 0 0) 0x7fb8a0137400 con 0x7fb8a00b4d60 2024-03-25T18:30:11.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.487+0000 7fb8977fe640 1 -- 172.21.15.155:0/4074560518 >> v1:172.21.15.167:6789/0 conn(0x7fb8a00b1730 legacy=0x7fb8a0134d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:11.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.487+0000 7fb8977fe640 1 -- 172.21.15.155:0/4074560518 >> v1:172.21.15.155:6789/0 conn(0x7fb8a014be90 legacy=0x7fb8a01e2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:11.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.487+0000 7fb8977fe640 10 monclient: _finish_hunting 0 2024-03-25T18:30:11.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.487+0000 7fb8977fe640 1 monclient: found mon.c 2024-03-25T18:30:11.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.487+0000 7fb8977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:11.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.487+0000 7fb8977fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:30:11.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.487+0000 7fb8977fe640 1 -- 172.21.15.155:0/4074560518 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8a0140510 con 0x7fb8a00b4d60 2024-03-25T18:30:11.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.487+0000 7fb8977fe640 10 monclient: _finish_auth 0 2024-03-25T18:30:11.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.487+0000 7fb8977fe640 10 monclient: _check_auth_tickets 2024-03-25T18:30:11.492 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.487+0000 7fb8977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:11.492 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.487+0000 7fb8a509d640 5 monclient: authenticate success, global_id 4475 2024-03-25T18:30:11.492 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.487+0000 7fb8a509d640 10 monclient: _renew_subs 2024-03-25T18:30:11.492 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.487+0000 7fb8a509d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:30:11.492 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.487+0000 7fb8a509d640 1 -- 172.21.15.155:0/4074560518 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8a0144bc0 con 0x7fb8a00b4d60 2024-03-25T18:30:11.492 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.487+0000 7fb8a509d640 10 client.4475.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:11.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.487+0000 7fb8a509d640 10 monclient: _renew_subs 2024-03-25T18:30:11.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.487+0000 7fb8a509d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:30:11.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.487+0000 7fb8a509d640 1 -- 172.21.15.155:0/4074560518 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8a0137e10 con 0x7fb8a00b4d60 2024-03-25T18:30:11.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.487+0000 7fb8a509d640 1 librados: init done 2024-03-25T18:30:11.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.488+0000 7fb8a509d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:11.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.488+0000 7fb8a509d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:11.494 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.488+0000 7fb8a509d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:30:11.494 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.488+0000 7fb8a509d640 1 -- 172.21.15.155:0/4074560518 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8a0144f60 con 0x7fb8a00b4d60 2024-03-25T18:30:11.825 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.823+0000 7fb8977fe640 1 -- 172.21.15.155:0/4074560518 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb8880047e0 con 0x7fb8a00b4d60 2024-03-25T18:30:11.825 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.823+0000 7fb8977fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:11.825 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.823+0000 7fb8977fe640 1 -- 172.21.15.155:0/4074560518 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fb888004c30 con 0x7fb8a00b4d60 2024-03-25T18:30:11.825 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.823+0000 7fb8977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:11.825 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.823+0000 7fb8977fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:30:11.825 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.823+0000 7fb8977fe640 10 monclient: dump: 2024-03-25T18:30:11.825 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:30:11.825 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:30:11.825 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:11.825 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:11.825 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:11.825 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:30:11.826 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:30:11.826 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:30:11.826 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:30:11.826 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:30:11.826 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.823+0000 7fb89dd9b640 4 set_mon_vals no callback set 2024-03-25T18:30:11.827 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.824+0000 7fb8977fe640 1 -- 172.21.15.155:0/4074560518 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7fb888004e50 con 0x7fb8a00b4d60 2024-03-25T18:30:11.827 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.824+0000 7fb89ed9d640 10 monclient: get_auth_request con 0x7fb88006ddd0 auth_method 0 2024-03-25T18:30:11.827 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.825+0000 7fb8977fe640 10 client.4475.objecter ms_handle_connect 0x7fb88006ddd0 2024-03-25T18:30:11.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.830+0000 7fb8977fe640 1 -- 172.21.15.155:0/4074560518 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7fb8880547e0 con 0x7fb8a00b4d60 2024-03-25T18:30:11.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.830+0000 7fb8977fe640 10 client.4475.objecter ms_dispatch 0x7fb8a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:30:11.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.830+0000 7fb8977fe640 3 client.4475.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:30:11.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.830+0000 7fb8977fe640 3 client.4475.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:30:11.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.830+0000 7fb8977fe640 20 client.4475.objecter dump_active .. 0 homeless 2024-03-25T18:30:11.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.830+0000 7fb8977fe640 1 -- 172.21.15.155:0/4074560518 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb8880502f0 con 0x7fb8a00b4d60 2024-03-25T18:30:11.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.830+0000 7fb8977fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:11.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:11.830+0000 7fb8977fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:12.086 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:12.085+0000 7fb8a509d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:12.087 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:12.085+0000 7fb8a509d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:12.087 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:12.085+0000 7fb8a509d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:30:12.087 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:12.085+0000 7fb8a509d640 1 -- 172.21.15.155:0/4074560518 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb8a0144d50 con 0x7fb8a00b4d60 2024-03-25T18:30:12.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:12.114+0000 7fb8977fe640 1 -- 172.21.15.155:0/4074560518 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7fb888052220 con 0x7fb8a00b4d60 2024-03-25T18:30:12.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:12.114+0000 7fb8977fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:12.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:12.114+0000 7fb8977fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:12.116 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:30:12.116 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:12.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:12.117+0000 7fb8a509d640 10 librados: watch_flush enter 2024-03-25T18:30:12.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:12.117+0000 7fb8a509d640 10 librados: watch_flush exit 2024-03-25T18:30:12.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:12.117+0000 7fb8a509d640 20 client.4475.objecter shutdown clearing up homeless session... 2024-03-25T18:30:12.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:12.117+0000 7fb8a509d640 10 client.4475.objecter successfully canceled tick 2024-03-25T18:30:12.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:12.117+0000 7fb8a509d640 1 -- 172.21.15.155:0/4074560518 >> v1:172.21.15.167:6800/2919070503 conn(0x7fb88006ddd0 legacy=0x7fb880070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:12.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:12.117+0000 7fb8a509d640 10 monclient: shutdown 2024-03-25T18:30:12.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:12.117+0000 7fb8a509d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:12.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:12.117+0000 7fb8a509d640 1 -- 172.21.15.155:0/4074560518 >> v1:172.21.15.155:6790/0 conn(0x7fb8a00b4d60 legacy=0x7fb8a01d7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:12.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:12.117+0000 7fb8a509d640 1 -- 172.21.15.155:0/4074560518 shutdown_connections 2024-03-25T18:30:12.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:12.117+0000 7fb8a509d640 1 -- 172.21.15.155:0/4074560518 >> 172.21.15.155:0/4074560518 conn(0x7fb8a00617a0 msgr2=0x7fb8a00b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:12.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:12.117+0000 7fb8a509d640 1 -- 172.21.15.155:0/4074560518 shutdown_connections 2024-03-25T18:30:12.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:12.117+0000 7fb8a509d640 1 -- 172.21.15.155:0/4074560518 wait complete. 2024-03-25T18:30:12.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:12.117+0000 7fb8a509d640 1 librados: shutdown 2024-03-25T18:30:12.130 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:18.937 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:19.052 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.049+0000 7f7b9cac1640 10 monclient: get_monmap_and_config 2024-03-25T18:30:19.052 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.049+0000 7f7b9cac1640 10 monclient: build_initial_monmap 2024-03-25T18:30:19.052 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.049+0000 7f7b9cac1640 10 monclient: monmap: 2024-03-25T18:30:19.052 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:30:19.052 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:30:19.052 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:30:19.051476+0000 2024-03-25T18:30:19.052 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:30:19.051476+0000 2024-03-25T18:30:19.052 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:19.052 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:30:19.052 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:30:19.052 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:30:19.052 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:30:19.053 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:30:19.053 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.050+0000 7f7b9cac1640 1 Processor -- start 2024-03-25T18:30:19.053 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.050+0000 7f7b9cac1640 1 -- start start 2024-03-25T18:30:19.053 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.050+0000 7f7b9cac1640 10 monclient: init 2024-03-25T18:30:19.054 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.050+0000 7f7b9cac1640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:19.054 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.050+0000 7f7b9cac1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:30:19.054 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.050+0000 7f7b9cac1640 10 monclient(hunting): picked mon.noname-c con 0x7f7b980a98f0 addr v1:172.21.15.167:6789/0 2024-03-25T18:30:19.054 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.050+0000 7f7b9cac1640 10 monclient(hunting): picked mon.noname-b con 0x7f7b9813f710 addr v1:172.21.15.155:6790/0 2024-03-25T18:30:19.054 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.050+0000 7f7b9cac1640 10 monclient(hunting): picked mon.noname-a con 0x7f7b981430c0 addr v1:172.21.15.155:6789/0 2024-03-25T18:30:19.054 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.050+0000 7f7b9cac1640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b98142520 con 0x7f7b981430c0 2024-03-25T18:30:19.054 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.051+0000 7f7b9cac1640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b980b28a0 con 0x7f7b980a98f0 2024-03-25T18:30:19.054 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.051+0000 7f7b9cac1640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b980b2a30 con 0x7f7b9813f710 2024-03-25T18:30:19.054 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.051+0000 7f7b9cac1640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:19.054 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.051+0000 7f7b9cac1640 10 monclient(hunting): authenticate will time out at 7865.361746s 2024-03-25T18:30:19.058 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.056+0000 7f7b9659c640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f7b980a98f0 0x7f7b980a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:34126/0 (socket says 172.21.15.155:34126) 2024-03-25T18:30:19.058 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.056+0000 7f7b9659c640 1 -- 172.21.15.155:0/1865820249 learned_addr learned my addr 172.21.15.155:0/1865820249 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:30:19.068 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.066+0000 7f7b9559a640 1 -- 172.21.15.155:0/1865820249 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 624771638 0 0) 0x7f7b980b28a0 con 0x7f7b980a98f0 2024-03-25T18:30:19.068 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.066+0000 7f7b9559a640 10 monclient(hunting): _negotiate 2024-03-25T18:30:19.068 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.066+0000 7f7b9559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:19.068 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.066+0000 7f7b9559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:19.068 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.066+0000 7f7b9559a640 10 monclient(hunting): my global_id is 4465 2024-03-25T18:30:19.068 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.066+0000 7f7b9559a640 1 -- 172.21.15.155:0/1865820249 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b80003520 con 0x7f7b980a98f0 2024-03-25T18:30:19.070 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.068+0000 7f7b9559a640 1 -- 172.21.15.155:0/1865820249 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1722667308 0 0) 0x7f7b980b2a30 con 0x7f7b9813f710 2024-03-25T18:30:19.070 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.068+0000 7f7b9559a640 10 monclient(hunting): _negotiate 2024-03-25T18:30:19.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.068+0000 7f7b9559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:19.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.068+0000 7f7b9559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:19.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.068+0000 7f7b9559a640 10 monclient(hunting): my global_id is 4478 2024-03-25T18:30:19.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.068+0000 7f7b9559a640 1 -- 172.21.15.155:0/1865820249 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b980b28a0 con 0x7f7b9813f710 2024-03-25T18:30:19.084 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.081+0000 7f7b9559a640 1 -- 172.21.15.155:0/1865820249 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 402486906 0 0) 0x7f7b980b28a0 con 0x7f7b9813f710 2024-03-25T18:30:19.084 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.081+0000 7f7b9559a640 1 -- 172.21.15.155:0/1865820249 >> v1:172.21.15.167:6789/0 conn(0x7f7b980a98f0 legacy=0x7f7b980a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:19.084 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.081+0000 7f7b9559a640 1 -- 172.21.15.155:0/1865820249 >> v1:172.21.15.155:6789/0 conn(0x7f7b981430c0 legacy=0x7f7b9814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:30:19.084 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.081+0000 7f7b9559a640 10 monclient: _finish_hunting 0 2024-03-25T18:30:19.084 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.081+0000 7f7b9559a640 1 monclient: found mon.noname-b 2024-03-25T18:30:19.084 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.081+0000 7f7b9559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:19.084 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.081+0000 7f7b9559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-25T18:30:19.084 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.081+0000 7f7b9559a640 1 -- 172.21.15.155:0/1865820249 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b980b44c0 con 0x7f7b9813f710 2024-03-25T18:30:19.084 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.081+0000 7f7b9559a640 10 monclient: _finish_auth 0 2024-03-25T18:30:19.084 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.081+0000 7f7b9559a640 10 monclient: _check_auth_tickets 2024-03-25T18:30:19.084 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.081+0000 7f7b9559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:19.084 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.081+0000 7f7b9559a640 1 -- 172.21.15.155:0/1865820249 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f7b8c001670 con 0x7f7b9813f710 2024-03-25T18:30:19.084 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.081+0000 7f7b9cac1640 5 monclient: authenticate success, global_id 4478 2024-03-25T18:30:19.085 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.081+0000 7f7b9cac1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:19.085 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.081+0000 7f7b9559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:19.085 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.081+0000 7f7b9559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:30:19.085 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.081+0000 7f7b9559a640 10 monclient: dump: 2024-03-25T18:30:19.085 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:30:19.085 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:30:19.085 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:19.085 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:19.085 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:19.085 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:30:19.085 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:30:19.085 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:30:19.085 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:30:19.085 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:30:19.086 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.081+0000 7f7b9cac1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:19.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.116+0000 7f7b9559a640 1 -- 172.21.15.155:0/1865820249 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b8c0017d0 con 0x7f7b9813f710 2024-03-25T18:30:19.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.116+0000 7f7b9559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:19.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.116+0000 7f7b9559a640 1 -- 172.21.15.155:0/1865820249 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f7b8c001c20 con 0x7f7b9813f710 2024-03-25T18:30:19.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.116+0000 7f7b9559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:19.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.116+0000 7f7b9559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:30:19.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.116+0000 7f7b9559a640 10 monclient: dump: 2024-03-25T18:30:19.119 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:30:19.119 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:30:19.119 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:19.119 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:19.119 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:19.119 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:30:19.119 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:30:19.119 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:30:19.119 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:30:19.119 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:30:19.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.116+0000 7f7b9cac1640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:19.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.116+0000 7f7b9cac1640 4 set_mon_vals no callback set 2024-03-25T18:30:19.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.117+0000 7f7b9cac1640 10 monclient: shutdown 2024-03-25T18:30:19.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.117+0000 7f7b9cac1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:19.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.117+0000 7f7b9cac1640 1 -- 172.21.15.155:0/1865820249 >> v1:172.21.15.155:6790/0 conn(0x7f7b9813f710 legacy=0x7f7b981429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:19.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.117+0000 7f7b9cac1640 1 -- 172.21.15.155:0/1865820249 shutdown_connections 2024-03-25T18:30:19.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.117+0000 7f7b9cac1640 1 -- 172.21.15.155:0/1865820249 >> 172.21.15.155:0/1865820249 conn(0x7f7b980b7ea0 msgr2=0x7f7b980b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:19.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.117+0000 7f7b9cac1640 1 -- 172.21.15.155:0/1865820249 shutdown_connections 2024-03-25T18:30:19.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.117+0000 7f7b9cac1640 1 -- 172.21.15.155:0/1865820249 wait complete. 2024-03-25T18:30:19.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.118+0000 7f7b9cac1640 10 monclient: build_initial_monmap 2024-03-25T18:30:19.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.118+0000 7f7b9cac1640 10 monclient: monmap: 2024-03-25T18:30:19.122 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:30:19.122 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:19.122 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:30:19.122 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:30:19.122 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:19.122 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:30:19.122 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:30:19.122 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:30:19.123 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:30:19.123 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:30:19.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.118+0000 7f7b9cac1640 1 librados: starting msgr at 2024-03-25T18:30:19.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.118+0000 7f7b9cac1640 1 librados: starting objecter 2024-03-25T18:30:19.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.118+0000 7f7b9cac1640 1 Processor -- start 2024-03-25T18:30:19.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.118+0000 7f7b9cac1640 1 -- start start 2024-03-25T18:30:19.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.118+0000 7f7b9cac1640 1 librados: setting wanted keys 2024-03-25T18:30:19.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.118+0000 7f7b9cac1640 1 librados: calling monclient init 2024-03-25T18:30:19.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.118+0000 7f7b9cac1640 10 monclient: init 2024-03-25T18:30:19.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.119+0000 7f7b9cac1640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:19.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.119+0000 7f7b9cac1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:30:19.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.119+0000 7f7b9cac1640 10 monclient(hunting): picked mon.noname-b con 0x7f7b980a98f0 addr v1:172.21.15.167:6789/0 2024-03-25T18:30:19.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.119+0000 7f7b9cac1640 10 monclient(hunting): picked mon.noname-a con 0x7f7b9813f710 addr v1:172.21.15.155:6789/0 2024-03-25T18:30:19.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.119+0000 7f7b9cac1640 10 monclient(hunting): picked mon.noname-c con 0x7f7b981430c0 addr v1:172.21.15.155:6790/0 2024-03-25T18:30:19.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.119+0000 7f7b9cac1640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b98138dd0 con 0x7f7b9813f710 2024-03-25T18:30:19.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.119+0000 7f7b9cac1640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b98139820 con 0x7f7b980a98f0 2024-03-25T18:30:19.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.119+0000 7f7b9cac1640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b981388b0 con 0x7f7b981430c0 2024-03-25T18:30:19.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.119+0000 7f7b9cac1640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:19.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.119+0000 7f7b9cac1640 10 monclient(hunting): authenticate will time out at 7865.430312s 2024-03-25T18:30:19.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.132+0000 7f7b95d9b640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f7b9813f710 0x7f7b98134de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:55012/0 (socket says 172.21.15.155:55012) 2024-03-25T18:30:19.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.132+0000 7f7b95d9b640 1 -- 172.21.15.155:0/2345059837 learned_addr learned my addr 172.21.15.155:0/2345059837 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:30:19.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.133+0000 7f7b86ffd640 10 client.?.objecter ms_handle_connect 0x7f7b9813f710 2024-03-25T18:30:19.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.133+0000 7f7b86ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:19.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.136+0000 7f7b86ffd640 10 client.?.objecter ms_handle_connect 0x7f7b980a98f0 2024-03-25T18:30:19.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.136+0000 7f7b86ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:19.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.136+0000 7f7b86ffd640 10 client.?.objecter ms_handle_connect 0x7f7b981430c0 2024-03-25T18:30:19.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.136+0000 7f7b86ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:19.150 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.147+0000 7f7b86ffd640 1 -- 172.21.15.155:0/2345059837 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4270477598 0 0) 0x7f7b98138dd0 con 0x7f7b9813f710 2024-03-25T18:30:19.150 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.147+0000 7f7b86ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:19.150 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.147+0000 7f7b86ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:19.151 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.147+0000 7f7b86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:19.151 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.147+0000 7f7b86ffd640 10 monclient(hunting): my global_id is 4473 2024-03-25T18:30:19.151 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.147+0000 7f7b86ffd640 1 -- 172.21.15.155:0/2345059837 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b78004eb0 con 0x7f7b9813f710 2024-03-25T18:30:19.168 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.166+0000 7f7b86ffd640 1 -- 172.21.15.155:0/2345059837 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3187378596 0 0) 0x7f7b78004eb0 con 0x7f7b9813f710 2024-03-25T18:30:19.168 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.166+0000 7f7b86ffd640 1 -- 172.21.15.155:0/2345059837 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b98138dd0 con 0x7f7b9813f710 2024-03-25T18:30:19.169 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.166+0000 7f7b86ffd640 1 -- 172.21.15.155:0/2345059837 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f7b8c001630 con 0x7f7b9813f710 2024-03-25T18:30:19.169 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.166+0000 7f7b86ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:19.169 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.166+0000 7f7b86ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:30:19.169 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.166+0000 7f7b86ffd640 10 monclient(hunting): dump: 2024-03-25T18:30:19.169 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:30:19.169 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:30:19.169 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:19.169 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:19.169 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:19.169 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:30:19.169 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:30:19.169 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:30:19.169 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:30:19.170 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:30:19.170 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.166+0000 7f7b86ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:19.238 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.235+0000 7f7b86ffd640 1 -- 172.21.15.155:0/2345059837 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3927339993 0 0) 0x7f7b98138dd0 con 0x7f7b9813f710 2024-03-25T18:30:19.238 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.235+0000 7f7b86ffd640 1 -- 172.21.15.155:0/2345059837 >> v1:172.21.15.155:6790/0 conn(0x7f7b981430c0 legacy=0x7f7b981e1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:19.238 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.235+0000 7f7b86ffd640 1 -- 172.21.15.155:0/2345059837 >> v1:172.21.15.167:6789/0 conn(0x7f7b980a98f0 legacy=0x7f7b98137780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:19.238 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.235+0000 7f7b86ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:30:19.238 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.235+0000 7f7b86ffd640 1 monclient: found mon.a 2024-03-25T18:30:19.238 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.235+0000 7f7b86ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:19.238 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.235+0000 7f7b86ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:30:19.238 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.235+0000 7f7b86ffd640 1 -- 172.21.15.155:0/2345059837 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b981354f0 con 0x7f7b9813f710 2024-03-25T18:30:19.238 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.235+0000 7f7b86ffd640 10 monclient: _finish_auth 0 2024-03-25T18:30:19.238 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.235+0000 7f7b86ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:30:19.239 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.235+0000 7f7b86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:19.239 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.235+0000 7f7b9cac1640 5 monclient: authenticate success, global_id 4473 2024-03-25T18:30:19.239 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.235+0000 7f7b9cac1640 10 monclient: _renew_subs 2024-03-25T18:30:19.239 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.235+0000 7f7b9cac1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:30:19.239 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.235+0000 7f7b9cac1640 1 -- 172.21.15.155:0/2345059837 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b98135740 con 0x7f7b9813f710 2024-03-25T18:30:19.239 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.235+0000 7f7b9cac1640 10 client.4473.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:19.239 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.235+0000 7f7b9cac1640 10 monclient: _renew_subs 2024-03-25T18:30:19.239 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.235+0000 7f7b9cac1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:30:19.239 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.235+0000 7f7b9cac1640 1 -- 172.21.15.155:0/2345059837 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b98138320 con 0x7f7b9813f710 2024-03-25T18:30:19.239 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.235+0000 7f7b9cac1640 1 librados: init done 2024-03-25T18:30:19.240 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.236+0000 7f7b9cac1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:19.240 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.236+0000 7f7b9cac1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:19.240 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.236+0000 7f7b9cac1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:30:19.240 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.236+0000 7f7b9cac1640 1 -- 172.21.15.155:0/2345059837 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b98138a40 con 0x7f7b9813f710 2024-03-25T18:30:19.542 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.541+0000 7f7b86ffd640 1 -- 172.21.15.155:0/2345059837 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b8c0038a0 con 0x7f7b9813f710 2024-03-25T18:30:19.542 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.541+0000 7f7b86ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:19.542 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.541+0000 7f7b86ffd640 1 -- 172.21.15.155:0/2345059837 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f7b8c003c70 con 0x7f7b9813f710 2024-03-25T18:30:19.542 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.541+0000 7f7b86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:19.542 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.541+0000 7f7b86ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:30:19.542 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.541+0000 7f7b86ffd640 10 monclient: dump: 2024-03-25T18:30:19.542 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:30:19.543 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:30:19.543 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:19.543 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:19.543 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:19.543 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:30:19.543 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:30:19.543 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:30:19.543 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:30:19.543 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:30:19.543 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.541+0000 7f7b94d99640 4 set_mon_vals no callback set 2024-03-25T18:30:19.544 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.541+0000 7f7b86ffd640 1 -- 172.21.15.155:0/2345059837 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7f7b8c003e90 con 0x7f7b9813f710 2024-03-25T18:30:19.544 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.542+0000 7f7b9659c640 10 monclient: get_auth_request con 0x7f7b7806dd00 auth_method 0 2024-03-25T18:30:19.544 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.542+0000 7f7b86ffd640 10 client.4473.objecter ms_handle_connect 0x7f7b7806dd00 2024-03-25T18:30:19.549 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.548+0000 7f7b86ffd640 1 -- 172.21.15.155:0/2345059837 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7f7b8c0549e0 con 0x7f7b9813f710 2024-03-25T18:30:19.550 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.548+0000 7f7b86ffd640 10 client.4473.objecter ms_dispatch 0x7f7b98000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:30:19.550 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.548+0000 7f7b86ffd640 3 client.4473.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:30:19.550 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.548+0000 7f7b86ffd640 3 client.4473.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:30:19.550 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.548+0000 7f7b86ffd640 20 client.4473.objecter dump_active .. 0 homeless 2024-03-25T18:30:19.550 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.548+0000 7f7b86ffd640 1 -- 172.21.15.155:0/2345059837 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7b8c0504f0 con 0x7f7b9813f710 2024-03-25T18:30:19.550 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.548+0000 7f7b86ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:19.550 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.548+0000 7f7b86ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:19.793 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.792+0000 7f7b9cac1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:19.793 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.792+0000 7f7b9cac1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:19.793 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.792+0000 7f7b9cac1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:30:19.794 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.792+0000 7f7b9cac1640 1 -- 172.21.15.155:0/2345059837 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7b980627b0 con 0x7f7b9813f710 2024-03-25T18:30:19.828 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.827+0000 7f7b86ffd640 1 -- 172.21.15.155:0/2345059837 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7f7b8c052420 con 0x7f7b9813f710 2024-03-25T18:30:19.828 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.827+0000 7f7b86ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:19.828 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.827+0000 7f7b86ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:19.829 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:30:19.829 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:19.830 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.830+0000 7f7b9cac1640 10 librados: watch_flush enter 2024-03-25T18:30:19.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.830+0000 7f7b9cac1640 10 librados: watch_flush exit 2024-03-25T18:30:19.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.830+0000 7f7b9cac1640 20 client.4473.objecter shutdown clearing up homeless session... 2024-03-25T18:30:19.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.830+0000 7f7b9cac1640 10 client.4473.objecter successfully canceled tick 2024-03-25T18:30:19.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.830+0000 7f7b9cac1640 1 -- 172.21.15.155:0/2345059837 >> v1:172.21.15.167:6800/2919070503 conn(0x7f7b7806dd00 legacy=0x7f7b780701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:19.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.830+0000 7f7b9cac1640 10 monclient: shutdown 2024-03-25T18:30:19.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.830+0000 7f7b9cac1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:19.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.830+0000 7f7b9cac1640 1 -- 172.21.15.155:0/2345059837 >> v1:172.21.15.155:6789/0 conn(0x7f7b9813f710 legacy=0x7f7b98134de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:19.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.830+0000 7f7b9cac1640 1 -- 172.21.15.155:0/2345059837 shutdown_connections 2024-03-25T18:30:19.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.830+0000 7f7b9cac1640 1 -- 172.21.15.155:0/2345059837 >> 172.21.15.155:0/2345059837 conn(0x7f7b980b7ea0 msgr2=0x7f7b98074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:19.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.830+0000 7f7b9cac1640 1 -- 172.21.15.155:0/2345059837 shutdown_connections 2024-03-25T18:30:19.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.830+0000 7f7b9cac1640 1 -- 172.21.15.155:0/2345059837 wait complete. 2024-03-25T18:30:19.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:19.830+0000 7f7b9cac1640 1 librados: shutdown 2024-03-25T18:30:19.843 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:26.453 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:06:00.057 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:26.463 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:06:00.067 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:26.499 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:06:00.103 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:26.506 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:06:00.084 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:26.510 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:06:00.088 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:26.517 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:06:00.121 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:26.522 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:06:00.127 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:26.538 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:06:00.116 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:26.565 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:06:00.170 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:26.651 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:26.759 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.758+0000 7fc78ca33640 10 monclient: get_monmap_and_config 2024-03-25T18:30:26.759 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.758+0000 7fc78ca33640 10 monclient: build_initial_monmap 2024-03-25T18:30:26.760 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.758+0000 7fc78ca33640 10 monclient: monmap: 2024-03-25T18:30:26.760 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:30:26.760 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:30:26.760 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:30:26.759433+0000 2024-03-25T18:30:26.760 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:30:26.759433+0000 2024-03-25T18:30:26.760 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:26.760 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:30:26.760 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:30:26.760 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:30:26.760 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:30:26.760 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:30:26.761 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.759+0000 7fc78ca33640 1 Processor -- start 2024-03-25T18:30:26.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.759+0000 7fc78ca33640 1 -- start start 2024-03-25T18:30:26.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.759+0000 7fc78ca33640 10 monclient: init 2024-03-25T18:30:26.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.760+0000 7fc78ca33640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:26.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.760+0000 7fc78ca33640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:30:26.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.760+0000 7fc78ca33640 10 monclient(hunting): picked mon.noname-c con 0x7fc7880a98f0 addr v1:172.21.15.167:6789/0 2024-03-25T18:30:26.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.760+0000 7fc78ca33640 10 monclient(hunting): picked mon.noname-b con 0x7fc78813f710 addr v1:172.21.15.155:6790/0 2024-03-25T18:30:26.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.760+0000 7fc78ca33640 10 monclient(hunting): picked mon.noname-a con 0x7fc7881430c0 addr v1:172.21.15.155:6789/0 2024-03-25T18:30:26.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.760+0000 7fc78ca33640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc788142520 con 0x7fc7881430c0 2024-03-25T18:30:26.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.760+0000 7fc78ca33640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7880b28a0 con 0x7fc7880a98f0 2024-03-25T18:30:26.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.760+0000 7fc78ca33640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7880b2a30 con 0x7fc78813f710 2024-03-25T18:30:26.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.760+0000 7fc78ca33640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:26.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.760+0000 7fc78ca33640 10 monclient(hunting): authenticate will time out at 7873.069946s 2024-03-25T18:30:26.773 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.772+0000 7fc78659c640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7fc7880a98f0 0x7fc7880a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:53868/0 (socket says 172.21.15.155:53868) 2024-03-25T18:30:26.773 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.772+0000 7fc78659c640 1 -- 172.21.15.155:0/3307891137 learned_addr learned my addr 172.21.15.155:0/3307891137 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:30:26.785 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.784+0000 7fc78559a640 1 -- 172.21.15.155:0/3307891137 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1816716596 0 0) 0x7fc7880b28a0 con 0x7fc7880a98f0 2024-03-25T18:30:26.785 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.784+0000 7fc78559a640 10 monclient(hunting): _negotiate 2024-03-25T18:30:26.785 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.784+0000 7fc78559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:26.785 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.784+0000 7fc78559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:26.785 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.784+0000 7fc78559a640 10 monclient(hunting): my global_id is 4471 2024-03-25T18:30:26.785 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.784+0000 7fc78559a640 1 -- 172.21.15.155:0/3307891137 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc770003520 con 0x7fc7880a98f0 2024-03-25T18:30:26.805 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.803+0000 7fc78559a640 1 -- 172.21.15.155:0/3307891137 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1362427443 0 0) 0x7fc770003520 con 0x7fc7880a98f0 2024-03-25T18:30:26.805 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.803+0000 7fc78559a640 1 -- 172.21.15.155:0/3307891137 >> v1:172.21.15.155:6790/0 conn(0x7fc78813f710 legacy=0x7fc7881429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:26.805 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.803+0000 7fc78559a640 1 -- 172.21.15.155:0/3307891137 >> v1:172.21.15.155:6789/0 conn(0x7fc7881430c0 legacy=0x7fc78814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:26.805 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.803+0000 7fc78559a640 10 monclient: _finish_hunting 0 2024-03-25T18:30:26.805 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.803+0000 7fc78559a640 1 monclient: found mon.noname-c 2024-03-25T18:30:26.805 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.803+0000 7fc78559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:26.805 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.803+0000 7fc78559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.167:6789/0 2024-03-25T18:30:26.805 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.803+0000 7fc78559a640 1 -- 172.21.15.155:0/3307891137 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7880b44c0 con 0x7fc7880a98f0 2024-03-25T18:30:26.805 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.803+0000 7fc78559a640 10 monclient: _finish_auth 0 2024-03-25T18:30:26.805 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.803+0000 7fc78559a640 10 monclient: _check_auth_tickets 2024-03-25T18:30:26.805 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.803+0000 7fc78559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:26.805 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.803+0000 7fc78559a640 1 -- 172.21.15.155:0/3307891137 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fc778001690 con 0x7fc7880a98f0 2024-03-25T18:30:26.806 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.803+0000 7fc78ca33640 5 monclient: authenticate success, global_id 4471 2024-03-25T18:30:26.806 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.803+0000 7fc78ca33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:26.806 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.803+0000 7fc78559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:26.806 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.803+0000 7fc78559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:30:26.806 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.803+0000 7fc78559a640 10 monclient: dump: 2024-03-25T18:30:26.806 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:30:26.806 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:30:26.806 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:26.806 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:26.806 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:26.806 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:30:26.806 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:30:26.806 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:30:26.806 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:30:26.807 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:30:26.807 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.803+0000 7fc78ca33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:26.825 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.824+0000 7fc78559a640 1 -- 172.21.15.155:0/3307891137 <== mon.1 v1:172.21.15.167:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc7780017f0 con 0x7fc7880a98f0 2024-03-25T18:30:26.826 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.824+0000 7fc78559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:26.826 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.824+0000 7fc78559a640 1 -- 172.21.15.155:0/3307891137 <== mon.1 v1:172.21.15.167:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fc778001c00 con 0x7fc7880a98f0 2024-03-25T18:30:26.826 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.824+0000 7fc78ca33640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:26.826 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.824+0000 7fc78ca33640 4 set_mon_vals no callback set 2024-03-25T18:30:26.826 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.824+0000 7fc78559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:26.826 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.824+0000 7fc78559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:26.826 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.825+0000 7fc78559a640 10 monclient: dump: 2024-03-25T18:30:26.826 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:30:26.826 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:30:26.826 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:26.826 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:26.826 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:26.827 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:30:26.827 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:30:26.827 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:30:26.827 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:30:26.827 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:30:26.829 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.825+0000 7fc78ca33640 10 monclient: shutdown 2024-03-25T18:30:26.829 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.825+0000 7fc78ca33640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:26.829 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.825+0000 7fc78ca33640 1 -- 172.21.15.155:0/3307891137 >> v1:172.21.15.167:6789/0 conn(0x7fc7880a98f0 legacy=0x7fc7880a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:26.829 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.825+0000 7fc78ca33640 1 -- 172.21.15.155:0/3307891137 shutdown_connections 2024-03-25T18:30:26.829 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.825+0000 7fc78ca33640 1 -- 172.21.15.155:0/3307891137 >> 172.21.15.155:0/3307891137 conn(0x7fc7880b7ea0 msgr2=0x7fc7880b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:26.829 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.825+0000 7fc78ca33640 1 -- 172.21.15.155:0/3307891137 shutdown_connections 2024-03-25T18:30:26.829 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.826+0000 7fc78ca33640 1 -- 172.21.15.155:0/3307891137 wait complete. 2024-03-25T18:30:26.830 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.826+0000 7fc78ca33640 10 monclient: build_initial_monmap 2024-03-25T18:30:26.830 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.826+0000 7fc78ca33640 10 monclient: monmap: 2024-03-25T18:30:26.830 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:30:26.830 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:26.830 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:30:26.830 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:30:26.830 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:26.830 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:30:26.830 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:30:26.830 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:30:26.830 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:30:26.830 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:30:26.830 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.827+0000 7fc78ca33640 1 librados: starting msgr at 2024-03-25T18:30:26.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.827+0000 7fc78ca33640 1 librados: starting objecter 2024-03-25T18:30:26.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.827+0000 7fc78ca33640 1 Processor -- start 2024-03-25T18:30:26.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.827+0000 7fc78ca33640 1 -- start start 2024-03-25T18:30:26.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.827+0000 7fc78ca33640 1 librados: setting wanted keys 2024-03-25T18:30:26.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.827+0000 7fc78ca33640 1 librados: calling monclient init 2024-03-25T18:30:26.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.827+0000 7fc78ca33640 10 monclient: init 2024-03-25T18:30:26.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.827+0000 7fc78ca33640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:26.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.827+0000 7fc78ca33640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:30:26.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.827+0000 7fc78ca33640 10 monclient(hunting): picked mon.noname-b con 0x7fc7880a98f0 addr v1:172.21.15.167:6789/0 2024-03-25T18:30:26.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.827+0000 7fc78ca33640 10 monclient(hunting): picked mon.noname-a con 0x7fc78813f710 addr v1:172.21.15.155:6789/0 2024-03-25T18:30:26.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.827+0000 7fc78ca33640 10 monclient(hunting): picked mon.noname-c con 0x7fc7881430c0 addr v1:172.21.15.155:6790/0 2024-03-25T18:30:26.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.827+0000 7fc78ca33640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc78813be70 con 0x7fc78813f710 2024-03-25T18:30:26.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.827+0000 7fc78ca33640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc788131660 con 0x7fc7880a98f0 2024-03-25T18:30:26.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.827+0000 7fc78ca33640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7881317f0 con 0x7fc7881430c0 2024-03-25T18:30:26.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.827+0000 7fc78ca33640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:26.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.827+0000 7fc78ca33640 10 monclient(hunting): authenticate will time out at 7873.137631s 2024-03-25T18:30:26.842 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.841+0000 7fc785d9b640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fc78813f710 0x7fc7881d76e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:54940/0 (socket says 172.21.15.155:54940) 2024-03-25T18:30:26.842 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.841+0000 7fc785d9b640 1 -- 172.21.15.155:0/101621627 learned_addr learned my addr 172.21.15.155:0/101621627 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:30:26.842 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.841+0000 7fc776ffd640 10 client.?.objecter ms_handle_connect 0x7fc78813f710 2024-03-25T18:30:26.843 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.842+0000 7fc776ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:26.851 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.850+0000 7fc776ffd640 10 client.?.objecter ms_handle_connect 0x7fc7881430c0 2024-03-25T18:30:26.851 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.850+0000 7fc776ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:26.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.852+0000 7fc776ffd640 10 client.?.objecter ms_handle_connect 0x7fc7880a98f0 2024-03-25T18:30:26.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.852+0000 7fc776ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:26.856 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.855+0000 7fc776ffd640 1 -- 172.21.15.155:0/101621627 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1026514244 0 0) 0x7fc78813be70 con 0x7fc78813f710 2024-03-25T18:30:26.857 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.855+0000 7fc776ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:26.857 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.855+0000 7fc776ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:26.857 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.855+0000 7fc776ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:26.857 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.855+0000 7fc776ffd640 10 monclient(hunting): my global_id is 4479 2024-03-25T18:30:26.857 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.855+0000 7fc776ffd640 1 -- 172.21.15.155:0/101621627 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc768004eb0 con 0x7fc78813f710 2024-03-25T18:30:26.866 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.865+0000 7fc776ffd640 1 -- 172.21.15.155:0/101621627 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 913198344 0 0) 0x7fc788131660 con 0x7fc7880a98f0 2024-03-25T18:30:26.866 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.865+0000 7fc776ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:26.866 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.865+0000 7fc776ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:26.866 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.865+0000 7fc776ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:26.867 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.865+0000 7fc776ffd640 10 monclient(hunting): my global_id is 4474 2024-03-25T18:30:26.867 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.865+0000 7fc776ffd640 1 -- 172.21.15.155:0/101621627 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc78813be70 con 0x7fc7880a98f0 2024-03-25T18:30:26.867 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.866+0000 7fc776ffd640 1 -- 172.21.15.155:0/101621627 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3957124814 0 0) 0x7fc7881317f0 con 0x7fc7881430c0 2024-03-25T18:30:26.867 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.866+0000 7fc776ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:26.867 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.866+0000 7fc776ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:26.867 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.866+0000 7fc776ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:26.867 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.866+0000 7fc776ffd640 10 monclient(hunting): my global_id is 4487 2024-03-25T18:30:26.868 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.866+0000 7fc776ffd640 1 -- 172.21.15.155:0/101621627 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc788131660 con 0x7fc7881430c0 2024-03-25T18:30:26.875 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.873+0000 7fc776ffd640 1 -- 172.21.15.155:0/101621627 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3831272445 0 0) 0x7fc768004eb0 con 0x7fc78813f710 2024-03-25T18:30:26.875 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.873+0000 7fc776ffd640 1 -- 172.21.15.155:0/101621627 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7881317f0 con 0x7fc78813f710 2024-03-25T18:30:26.875 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.873+0000 7fc776ffd640 1 -- 172.21.15.155:0/101621627 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fc77c001a30 con 0x7fc78813f710 2024-03-25T18:30:26.875 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.873+0000 7fc776ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:26.875 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.873+0000 7fc776ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:30:26.875 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.873+0000 7fc776ffd640 10 monclient(hunting): dump: 2024-03-25T18:30:26.875 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:30:26.875 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:30:26.875 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:26.875 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:26.875 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:26.875 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:30:26.875 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:30:26.876 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:30:26.876 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:30:26.876 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:30:26.876 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.873+0000 7fc776ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:26.887 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.885+0000 7fc776ffd640 1 -- 172.21.15.155:0/101621627 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1514330456 0 0) 0x7fc7881317f0 con 0x7fc78813f710 2024-03-25T18:30:26.887 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.885+0000 7fc776ffd640 1 -- 172.21.15.155:0/101621627 >> v1:172.21.15.155:6790/0 conn(0x7fc7881430c0 legacy=0x7fc7881e2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:26.887 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.885+0000 7fc776ffd640 1 -- 172.21.15.155:0/101621627 >> v1:172.21.15.167:6789/0 conn(0x7fc7880a98f0 legacy=0x7fc788130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:26.887 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.885+0000 7fc776ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:30:26.887 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.885+0000 7fc776ffd640 1 monclient: found mon.a 2024-03-25T18:30:26.887 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.885+0000 7fc776ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:26.887 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.885+0000 7fc776ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:30:26.887 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.885+0000 7fc776ffd640 1 -- 172.21.15.155:0/101621627 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc788131980 con 0x7fc78813f710 2024-03-25T18:30:26.887 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.885+0000 7fc776ffd640 10 monclient: _finish_auth 0 2024-03-25T18:30:26.887 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.885+0000 7fc776ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:30:26.887 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.885+0000 7fc776ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:26.888 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.885+0000 7fc78ca33640 5 monclient: authenticate success, global_id 4479 2024-03-25T18:30:26.888 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.885+0000 7fc78ca33640 10 monclient: _renew_subs 2024-03-25T18:30:26.888 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.885+0000 7fc78ca33640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:30:26.888 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.885+0000 7fc78ca33640 1 -- 172.21.15.155:0/101621627 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc788135c60 con 0x7fc78813f710 2024-03-25T18:30:26.888 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.885+0000 7fc78ca33640 10 client.4479.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:26.888 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.885+0000 7fc78ca33640 10 monclient: _renew_subs 2024-03-25T18:30:26.888 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.885+0000 7fc78ca33640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:30:26.888 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.885+0000 7fc78ca33640 1 -- 172.21.15.155:0/101621627 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7880b7770 con 0x7fc78813f710 2024-03-25T18:30:26.888 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.885+0000 7fc78ca33640 1 librados: init done 2024-03-25T18:30:26.889 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.886+0000 7fc78ca33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:26.889 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.886+0000 7fc78ca33640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:26.889 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.887+0000 7fc78ca33640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:30:26.889 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:26.887+0000 7fc78ca33640 1 -- 172.21.15.155:0/101621627 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7881371d0 con 0x7fc78813f710 2024-03-25T18:30:27.198 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:27.197+0000 7fc776ffd640 1 -- 172.21.15.155:0/101621627 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc77c001c70 con 0x7fc78813f710 2024-03-25T18:30:27.198 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:27.197+0000 7fc776ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:27.199 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:27.197+0000 7fc776ffd640 1 -- 172.21.15.155:0/101621627 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fc77c003a60 con 0x7fc78813f710 2024-03-25T18:30:27.199 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:27.197+0000 7fc776ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:27.199 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:27.197+0000 7fc776ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:30:27.199 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:27.197+0000 7fc776ffd640 10 monclient: dump: 2024-03-25T18:30:27.199 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:30:27.199 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:30:27.199 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:27.199 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:27.199 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:27.199 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:30:27.199 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:30:27.199 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:30:27.199 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:30:27.199 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:30:27.199 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:27.197+0000 7fc784d99640 4 set_mon_vals no callback set 2024-03-25T18:30:27.200 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:27.198+0000 7fc776ffd640 1 -- 172.21.15.155:0/101621627 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7fc77c003c40 con 0x7fc78813f710 2024-03-25T18:30:27.200 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:27.198+0000 7fc78659c640 10 monclient: get_auth_request con 0x7fc76806de80 auth_method 0 2024-03-25T18:30:27.200 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:27.199+0000 7fc776ffd640 10 client.4479.objecter ms_handle_connect 0x7fc76806de80 2024-03-25T18:30:27.205 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:27.204+0000 7fc776ffd640 1 -- 172.21.15.155:0/101621627 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7fc77c054900 con 0x7fc78813f710 2024-03-25T18:30:27.205 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:27.204+0000 7fc776ffd640 10 client.4479.objecter ms_dispatch 0x7fc788000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:30:27.205 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:27.204+0000 7fc776ffd640 3 client.4479.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:30:27.206 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:27.204+0000 7fc776ffd640 3 client.4479.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:30:27.206 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:27.204+0000 7fc776ffd640 20 client.4479.objecter dump_active .. 0 homeless 2024-03-25T18:30:27.206 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:27.204+0000 7fc776ffd640 1 -- 172.21.15.155:0/101621627 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc77c050240 con 0x7fc78813f710 2024-03-25T18:30:27.206 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:27.204+0000 7fc776ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:27.206 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:27.204+0000 7fc776ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:27.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:27.458+0000 7fc78ca33640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:27.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:27.458+0000 7fc78ca33640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:27.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:27.458+0000 7fc78ca33640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:30:27.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:27.458+0000 7fc78ca33640 1 -- 172.21.15.155:0/101621627 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc7880627b0 con 0x7fc78813f710 2024-03-25T18:30:27.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:27.490+0000 7fc776ffd640 1 -- 172.21.15.155:0/101621627 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7fc77c052340 con 0x7fc78813f710 2024-03-25T18:30:27.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:27.490+0000 7fc776ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:27.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:27.490+0000 7fc776ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:27.491 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:30:27.491 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:27.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:27.492+0000 7fc78ca33640 10 librados: watch_flush enter 2024-03-25T18:30:27.494 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:27.492+0000 7fc78ca33640 10 librados: watch_flush exit 2024-03-25T18:30:27.494 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:27.492+0000 7fc78ca33640 20 client.4479.objecter shutdown clearing up homeless session... 2024-03-25T18:30:27.494 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:27.492+0000 7fc78ca33640 10 client.4479.objecter successfully canceled tick 2024-03-25T18:30:27.494 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:27.492+0000 7fc78ca33640 1 -- 172.21.15.155:0/101621627 >> v1:172.21.15.167:6800/2919070503 conn(0x7fc76806de80 legacy=0x7fc768070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:27.494 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:27.492+0000 7fc78ca33640 10 monclient: shutdown 2024-03-25T18:30:27.494 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:27.492+0000 7fc78ca33640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:27.494 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:27.492+0000 7fc78ca33640 1 -- 172.21.15.155:0/101621627 >> v1:172.21.15.155:6789/0 conn(0x7fc78813f710 legacy=0x7fc7881d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:27.494 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:27.492+0000 7fc78ca33640 1 -- 172.21.15.155:0/101621627 shutdown_connections 2024-03-25T18:30:27.494 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:27.492+0000 7fc78ca33640 1 -- 172.21.15.155:0/101621627 >> 172.21.15.155:0/101621627 conn(0x7fc7880b7ea0 msgr2=0x7fc788074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:27.494 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:27.492+0000 7fc78ca33640 1 -- 172.21.15.155:0/101621627 shutdown_connections 2024-03-25T18:30:27.494 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:27.492+0000 7fc78ca33640 1 -- 172.21.15.155:0/101621627 wait complete. 2024-03-25T18:30:27.494 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:27.492+0000 7fc78ca33640 1 librados: shutdown 2024-03-25T18:30:27.503 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:34.313 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:34.430 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.428+0000 7fdc4b49a640 10 monclient: get_monmap_and_config 2024-03-25T18:30:34.430 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.428+0000 7fdc4b49a640 10 monclient: build_initial_monmap 2024-03-25T18:30:34.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.429+0000 7fdc4b49a640 10 monclient: monmap: 2024-03-25T18:30:34.431 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:30:34.431 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:30:34.431 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:30:34.429961+0000 2024-03-25T18:30:34.431 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:30:34.429961+0000 2024-03-25T18:30:34.431 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:34.431 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:30:34.431 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:30:34.431 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:30:34.431 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:30:34.431 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:30:34.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.429+0000 7fdc4b49a640 1 Processor -- start 2024-03-25T18:30:34.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.430+0000 7fdc4b49a640 1 -- start start 2024-03-25T18:30:34.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.430+0000 7fdc4b49a640 10 monclient: init 2024-03-25T18:30:34.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.430+0000 7fdc4b49a640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:34.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.430+0000 7fdc4b49a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:30:34.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.430+0000 7fdc4b49a640 10 monclient(hunting): picked mon.noname-c con 0x7fdc440a98f0 addr v1:172.21.15.167:6789/0 2024-03-25T18:30:34.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.430+0000 7fdc4b49a640 10 monclient(hunting): picked mon.noname-a con 0x7fdc4413f710 addr v1:172.21.15.155:6789/0 2024-03-25T18:30:34.434 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.430+0000 7fdc4b49a640 10 monclient(hunting): picked mon.noname-b con 0x7fdc441430c0 addr v1:172.21.15.155:6790/0 2024-03-25T18:30:34.434 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.430+0000 7fdc4b49a640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc44142520 con 0x7fdc4413f710 2024-03-25T18:30:34.434 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.431+0000 7fdc4b49a640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc440b28a0 con 0x7fdc440a98f0 2024-03-25T18:30:34.434 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.431+0000 7fdc4b49a640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc440b2a30 con 0x7fdc441430c0 2024-03-25T18:30:34.434 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.431+0000 7fdc4b49a640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:34.434 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.431+0000 7fdc4b49a640 10 monclient(hunting): authenticate will time out at 7880.740854s 2024-03-25T18:30:34.434 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.433+0000 7fdc49a37640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fdc441430c0 0x7fdc4414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:53556/0 (socket says 172.21.15.155:53556) 2024-03-25T18:30:34.434 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.433+0000 7fdc49a37640 1 -- 172.21.15.155:0/1782858443 learned_addr learned my addr 172.21.15.155:0/1782858443 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:30:34.444 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.443+0000 7fdc3bfff640 1 -- 172.21.15.155:0/1782858443 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1944477442 0 0) 0x7fdc440b2a30 con 0x7fdc441430c0 2024-03-25T18:30:34.444 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.443+0000 7fdc3bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:30:34.444 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.443+0000 7fdc3bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:34.444 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.443+0000 7fdc3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:34.444 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.443+0000 7fdc3bfff640 10 monclient(hunting): my global_id is 4490 2024-03-25T18:30:34.444 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.443+0000 7fdc3bfff640 1 -- 172.21.15.155:0/1782858443 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc30003520 con 0x7fdc441430c0 2024-03-25T18:30:34.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.453+0000 7fdc3bfff640 1 -- 172.21.15.155:0/1782858443 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1721292171 0 0) 0x7fdc44142520 con 0x7fdc4413f710 2024-03-25T18:30:34.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.453+0000 7fdc3bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:30:34.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.453+0000 7fdc3bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:34.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.453+0000 7fdc3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:34.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.453+0000 7fdc3bfff640 10 monclient(hunting): my global_id is 4482 2024-03-25T18:30:34.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.453+0000 7fdc3bfff640 1 -- 172.21.15.155:0/1782858443 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc440b2a30 con 0x7fdc4413f710 2024-03-25T18:30:34.464 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.463+0000 7fdc3bfff640 1 -- 172.21.15.155:0/1782858443 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 868645261 0 0) 0x7fdc440b28a0 con 0x7fdc440a98f0 2024-03-25T18:30:34.464 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.463+0000 7fdc3bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:30:34.464 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.463+0000 7fdc3bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:34.464 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.463+0000 7fdc3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:34.464 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.463+0000 7fdc3bfff640 10 monclient(hunting): my global_id is 4477 2024-03-25T18:30:34.464 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.463+0000 7fdc3bfff640 1 -- 172.21.15.155:0/1782858443 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc44142520 con 0x7fdc440a98f0 2024-03-25T18:30:34.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.465+0000 7fdc3bfff640 1 -- 172.21.15.155:0/1782858443 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2787731393 0 0) 0x7fdc30003520 con 0x7fdc441430c0 2024-03-25T18:30:34.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.465+0000 7fdc3bfff640 1 -- 172.21.15.155:0/1782858443 >> v1:172.21.15.167:6789/0 conn(0x7fdc440a98f0 legacy=0x7fdc440a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:34.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.465+0000 7fdc3bfff640 1 -- 172.21.15.155:0/1782858443 >> v1:172.21.15.155:6789/0 conn(0x7fdc4413f710 legacy=0x7fdc441429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:34.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.465+0000 7fdc3bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:30:34.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.465+0000 7fdc3bfff640 1 monclient: found mon.noname-b 2024-03-25T18:30:34.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.465+0000 7fdc3bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:34.467 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.465+0000 7fdc3bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-25T18:30:34.467 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.465+0000 7fdc3bfff640 1 -- 172.21.15.155:0/1782858443 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc440b44c0 con 0x7fdc441430c0 2024-03-25T18:30:34.467 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.465+0000 7fdc3bfff640 10 monclient: _finish_auth 0 2024-03-25T18:30:34.467 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.465+0000 7fdc3bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:30:34.467 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.465+0000 7fdc3bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:34.467 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.465+0000 7fdc3bfff640 1 -- 172.21.15.155:0/1782858443 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fdc3c001a10 con 0x7fdc441430c0 2024-03-25T18:30:34.467 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.465+0000 7fdc3bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:34.467 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.465+0000 7fdc3bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:30:34.467 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.465+0000 7fdc3bfff640 10 monclient: dump: 2024-03-25T18:30:34.467 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:30:34.467 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:30:34.467 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:34.467 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:34.467 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:34.467 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:30:34.468 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:30:34.468 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:30:34.468 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:30:34.468 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:30:34.468 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.465+0000 7fdc4b49a640 5 monclient: authenticate success, global_id 4490 2024-03-25T18:30:34.468 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.465+0000 7fdc4b49a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:34.480 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.479+0000 7fdc3bfff640 1 -- 172.21.15.155:0/1782858443 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdc3c001b70 con 0x7fdc441430c0 2024-03-25T18:30:34.480 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.479+0000 7fdc3bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:34.480 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.479+0000 7fdc3bfff640 1 -- 172.21.15.155:0/1782858443 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fdc3c001f80 con 0x7fdc441430c0 2024-03-25T18:30:34.480 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.479+0000 7fdc3bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:34.480 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.479+0000 7fdc3bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:30:34.480 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.479+0000 7fdc3bfff640 10 monclient: dump: 2024-03-25T18:30:34.480 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:30:34.480 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:30:34.481 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:34.481 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:34.481 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:34.481 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:30:34.481 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:30:34.481 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:30:34.481 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:30:34.481 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:30:34.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.479+0000 7fdc4b49a640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:34.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.479+0000 7fdc4b49a640 4 set_mon_vals no callback set 2024-03-25T18:30:34.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.480+0000 7fdc4b49a640 10 monclient: shutdown 2024-03-25T18:30:34.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.480+0000 7fdc4b49a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:34.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.480+0000 7fdc4b49a640 1 -- 172.21.15.155:0/1782858443 >> v1:172.21.15.155:6790/0 conn(0x7fdc441430c0 legacy=0x7fdc4414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:34.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.480+0000 7fdc4b49a640 1 -- 172.21.15.155:0/1782858443 shutdown_connections 2024-03-25T18:30:34.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.480+0000 7fdc4b49a640 1 -- 172.21.15.155:0/1782858443 >> 172.21.15.155:0/1782858443 conn(0x7fdc440b7ea0 msgr2=0x7fdc440b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:34.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.480+0000 7fdc4b49a640 1 -- 172.21.15.155:0/1782858443 shutdown_connections 2024-03-25T18:30:34.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.480+0000 7fdc4b49a640 1 -- 172.21.15.155:0/1782858443 wait complete. 2024-03-25T18:30:34.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.481+0000 7fdc4b49a640 10 monclient: build_initial_monmap 2024-03-25T18:30:34.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.481+0000 7fdc4b49a640 10 monclient: monmap: 2024-03-25T18:30:34.483 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:30:34.483 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:34.483 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:30:34.483 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:30:34.484 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:34.484 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:30:34.484 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:30:34.484 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:30:34.484 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:30:34.484 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:30:34.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.481+0000 7fdc4b49a640 1 librados: starting msgr at 2024-03-25T18:30:34.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.481+0000 7fdc4b49a640 1 librados: starting objecter 2024-03-25T18:30:34.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.481+0000 7fdc4b49a640 1 Processor -- start 2024-03-25T18:30:34.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.481+0000 7fdc4b49a640 1 -- start start 2024-03-25T18:30:34.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.481+0000 7fdc4b49a640 1 librados: setting wanted keys 2024-03-25T18:30:34.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.481+0000 7fdc4b49a640 1 librados: calling monclient init 2024-03-25T18:30:34.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.481+0000 7fdc4b49a640 10 monclient: init 2024-03-25T18:30:34.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.482+0000 7fdc4b49a640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:34.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.482+0000 7fdc4b49a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:30:34.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.482+0000 7fdc4b49a640 10 monclient(hunting): picked mon.noname-b con 0x7fdc440a98f0 addr v1:172.21.15.167:6789/0 2024-03-25T18:30:34.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.482+0000 7fdc4b49a640 10 monclient(hunting): picked mon.noname-c con 0x7fdc4413f710 addr v1:172.21.15.155:6790/0 2024-03-25T18:30:34.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.482+0000 7fdc4b49a640 10 monclient(hunting): picked mon.noname-a con 0x7fdc441430c0 addr v1:172.21.15.155:6789/0 2024-03-25T18:30:34.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.482+0000 7fdc4b49a640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc44137740 con 0x7fdc441430c0 2024-03-25T18:30:34.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.482+0000 7fdc4b49a640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc44136c50 con 0x7fdc440a98f0 2024-03-25T18:30:34.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.482+0000 7fdc4b49a640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc44138fc0 con 0x7fdc4413f710 2024-03-25T18:30:34.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.482+0000 7fdc4b49a640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:34.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.482+0000 7fdc4b49a640 10 monclient(hunting): authenticate will time out at 7880.792433s 2024-03-25T18:30:34.492 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.491+0000 7fdc49236640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7fdc440a98f0 0x7fdc44137030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:58530/0 (socket says 172.21.15.155:58530) 2024-03-25T18:30:34.492 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.491+0000 7fdc49236640 1 -- 172.21.15.155:0/1589008282 learned_addr learned my addr 172.21.15.155:0/1589008282 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:30:34.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.492+0000 7fdc39ffb640 10 client.?.objecter ms_handle_connect 0x7fdc440a98f0 2024-03-25T18:30:34.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.492+0000 7fdc39ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:34.497 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.496+0000 7fdc39ffb640 1 -- 172.21.15.155:0/1589008282 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1937776827 0 0) 0x7fdc44136c50 con 0x7fdc440a98f0 2024-03-25T18:30:34.497 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.496+0000 7fdc39ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:30:34.497 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.496+0000 7fdc39ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:34.498 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.496+0000 7fdc39ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:34.498 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.496+0000 7fdc39ffb640 10 monclient(hunting): my global_id is 4480 2024-03-25T18:30:34.498 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.496+0000 7fdc39ffb640 1 -- 172.21.15.155:0/1589008282 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc28004eb0 con 0x7fdc440a98f0 2024-03-25T18:30:34.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.498+0000 7fdc39ffb640 10 client.?.objecter ms_handle_connect 0x7fdc441430c0 2024-03-25T18:30:34.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.498+0000 7fdc39ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:34.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.507+0000 7fdc39ffb640 10 client.?.objecter ms_handle_connect 0x7fdc4413f710 2024-03-25T18:30:34.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.507+0000 7fdc39ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:34.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.508+0000 7fdc39ffb640 1 -- 172.21.15.155:0/1589008282 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 290749782 0 0) 0x7fdc28004eb0 con 0x7fdc440a98f0 2024-03-25T18:30:34.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.508+0000 7fdc39ffb640 1 -- 172.21.15.155:0/1589008282 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdc44136c50 con 0x7fdc440a98f0 2024-03-25T18:30:34.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.508+0000 7fdc39ffb640 1 -- 172.21.15.155:0/1589008282 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fdc40001a10 con 0x7fdc440a98f0 2024-03-25T18:30:34.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.508+0000 7fdc39ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:34.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.508+0000 7fdc39ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:30:34.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.508+0000 7fdc39ffb640 10 monclient(hunting): dump: 2024-03-25T18:30:34.510 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:30:34.510 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:30:34.510 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:34.510 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:34.510 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:34.510 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:30:34.510 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:30:34.511 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:30:34.511 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:30:34.511 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:30:34.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.508+0000 7fdc39ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:34.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.512+0000 7fdc39ffb640 1 -- 172.21.15.155:0/1589008282 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4191870878 0 0) 0x7fdc44137740 con 0x7fdc441430c0 2024-03-25T18:30:34.514 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.512+0000 7fdc39ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:30:34.514 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.512+0000 7fdc39ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:34.514 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.512+0000 7fdc39ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:34.514 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.512+0000 7fdc39ffb640 10 monclient(hunting): my global_id is 4485 2024-03-25T18:30:34.514 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.512+0000 7fdc39ffb640 1 -- 172.21.15.155:0/1589008282 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc28004eb0 con 0x7fdc441430c0 2024-03-25T18:30:34.521 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.520+0000 7fdc39ffb640 1 -- 172.21.15.155:0/1589008282 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3004838850 0 0) 0x7fdc44138fc0 con 0x7fdc4413f710 2024-03-25T18:30:34.521 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.520+0000 7fdc39ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:30:34.521 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.520+0000 7fdc39ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:34.521 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.520+0000 7fdc39ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:34.521 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.520+0000 7fdc39ffb640 10 monclient(hunting): my global_id is 4493 2024-03-25T18:30:34.521 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.520+0000 7fdc39ffb640 1 -- 172.21.15.155:0/1589008282 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc44137740 con 0x7fdc4413f710 2024-03-25T18:30:34.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.523+0000 7fdc39ffb640 1 -- 172.21.15.155:0/1589008282 <== mon.1 v1:172.21.15.167:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1000180957 0 0) 0x7fdc44136c50 con 0x7fdc440a98f0 2024-03-25T18:30:34.525 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.523+0000 7fdc39ffb640 1 -- 172.21.15.155:0/1589008282 >> v1:172.21.15.155:6790/0 conn(0x7fdc4413f710 legacy=0x7fdc44136540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:34.525 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.523+0000 7fdc39ffb640 1 -- 172.21.15.155:0/1589008282 >> v1:172.21.15.155:6789/0 conn(0x7fdc441430c0 legacy=0x7fdc441e1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:34.525 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.523+0000 7fdc39ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:30:34.525 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.523+0000 7fdc39ffb640 1 monclient: found mon.b 2024-03-25T18:30:34.525 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.523+0000 7fdc39ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:34.525 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.523+0000 7fdc39ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:30:34.525 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.523+0000 7fdc39ffb640 1 -- 172.21.15.155:0/1589008282 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc44139150 con 0x7fdc440a98f0 2024-03-25T18:30:34.525 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.523+0000 7fdc39ffb640 10 monclient: _finish_auth 0 2024-03-25T18:30:34.525 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.523+0000 7fdc39ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:30:34.525 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.523+0000 7fdc39ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:34.525 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.523+0000 7fdc4b49a640 5 monclient: authenticate success, global_id 4480 2024-03-25T18:30:34.525 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.523+0000 7fdc4b49a640 10 monclient: _renew_subs 2024-03-25T18:30:34.525 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.523+0000 7fdc4b49a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:30:34.525 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.523+0000 7fdc4b49a640 1 -- 172.21.15.155:0/1589008282 --> v1:172.21.15.167:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc4413b2e0 con 0x7fdc440a98f0 2024-03-25T18:30:34.526 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.523+0000 7fdc4b49a640 10 client.4480.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:34.526 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.523+0000 7fdc4b49a640 10 monclient: _renew_subs 2024-03-25T18:30:34.526 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.523+0000 7fdc4b49a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:30:34.526 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.523+0000 7fdc4b49a640 1 -- 172.21.15.155:0/1589008282 --> v1:172.21.15.167:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc441395e0 con 0x7fdc440a98f0 2024-03-25T18:30:34.526 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.523+0000 7fdc4b49a640 1 librados: init done 2024-03-25T18:30:34.526 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.525+0000 7fdc4b49a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:34.526 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.525+0000 7fdc4b49a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:34.526 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.525+0000 7fdc4b49a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:30:34.527 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.525+0000 7fdc4b49a640 1 -- 172.21.15.155:0/1589008282 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc44136de0 con 0x7fdc440a98f0 2024-03-25T18:30:34.817 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.816+0000 7fdc39ffb640 1 -- 172.21.15.155:0/1589008282 <== mon.1 v1:172.21.15.167:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdc40001cb0 con 0x7fdc440a98f0 2024-03-25T18:30:34.817 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.816+0000 7fdc39ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:34.817 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.816+0000 7fdc39ffb640 1 -- 172.21.15.155:0/1589008282 <== mon.1 v1:172.21.15.167:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fdc40003a60 con 0x7fdc440a98f0 2024-03-25T18:30:34.817 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.816+0000 7fdc39ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:34.818 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.816+0000 7fdc39ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:34.818 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.816+0000 7fdc39ffb640 10 monclient: dump: 2024-03-25T18:30:34.818 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:30:34.818 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:30:34.818 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:34.818 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:34.818 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:34.818 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:30:34.818 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:30:34.818 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:30:34.818 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:30:34.819 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:30:34.819 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.816+0000 7fdc3b7fe640 4 set_mon_vals no callback set 2024-03-25T18:30:34.819 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.817+0000 7fdc39ffb640 1 -- 172.21.15.155:0/1589008282 <== mon.1 v1:172.21.15.167:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7fdc40003c80 con 0x7fdc440a98f0 2024-03-25T18:30:34.819 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.817+0000 7fdc48a35640 10 monclient: get_auth_request con 0x7fdc2806de80 auth_method 0 2024-03-25T18:30:34.819 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.818+0000 7fdc39ffb640 10 client.4480.objecter ms_handle_connect 0x7fdc2806de80 2024-03-25T18:30:34.824 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.822+0000 7fdc39ffb640 1 -- 172.21.15.155:0/1589008282 <== mon.1 v1:172.21.15.167:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7fdc40054890 con 0x7fdc440a98f0 2024-03-25T18:30:34.824 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.822+0000 7fdc39ffb640 10 client.4480.objecter ms_dispatch 0x7fdc44000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:30:34.824 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.822+0000 7fdc39ffb640 3 client.4480.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:30:34.824 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.822+0000 7fdc39ffb640 3 client.4480.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:30:34.824 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.822+0000 7fdc39ffb640 20 client.4480.objecter dump_active .. 0 homeless 2024-03-25T18:30:34.824 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.823+0000 7fdc39ffb640 1 -- 172.21.15.155:0/1589008282 <== mon.1 v1:172.21.15.167:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdc400503a0 con 0x7fdc440a98f0 2024-03-25T18:30:34.824 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.823+0000 7fdc39ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:34.824 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:34.823+0000 7fdc39ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:35.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:35.070+0000 7fdc4b49a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:35.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:35.070+0000 7fdc4b49a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:35.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:35.070+0000 7fdc4b49a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:30:35.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:35.070+0000 7fdc4b49a640 1 -- 172.21.15.155:0/1589008282 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdc440627b0 con 0x7fdc440a98f0 2024-03-25T18:30:35.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:35.095+0000 7fdc39ffb640 1 -- 172.21.15.155:0/1589008282 <== mon.1 v1:172.21.15.167:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7fdc400522d0 con 0x7fdc440a98f0 2024-03-25T18:30:35.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:35.095+0000 7fdc39ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:35.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:35.095+0000 7fdc39ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:35.097 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:30:35.097 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:35.100 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:35.098+0000 7fdc4b49a640 10 librados: watch_flush enter 2024-03-25T18:30:35.100 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:35.098+0000 7fdc4b49a640 10 librados: watch_flush exit 2024-03-25T18:30:35.100 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:35.098+0000 7fdc4b49a640 20 client.4480.objecter shutdown clearing up homeless session... 2024-03-25T18:30:35.100 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:35.098+0000 7fdc4b49a640 10 client.4480.objecter successfully canceled tick 2024-03-25T18:30:35.100 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:35.098+0000 7fdc4b49a640 1 -- 172.21.15.155:0/1589008282 >> v1:172.21.15.167:6800/2919070503 conn(0x7fdc2806de80 legacy=0x7fdc28070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:35.101 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:35.098+0000 7fdc4b49a640 10 monclient: shutdown 2024-03-25T18:30:35.101 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:35.098+0000 7fdc4b49a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:35.101 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:35.098+0000 7fdc4b49a640 1 -- 172.21.15.155:0/1589008282 >> v1:172.21.15.167:6789/0 conn(0x7fdc440a98f0 legacy=0x7fdc44137030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:35.101 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:35.099+0000 7fdc4b49a640 1 -- 172.21.15.155:0/1589008282 shutdown_connections 2024-03-25T18:30:35.101 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:35.099+0000 7fdc4b49a640 1 -- 172.21.15.155:0/1589008282 >> 172.21.15.155:0/1589008282 conn(0x7fdc440b7ea0 msgr2=0x7fdc44074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:35.101 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:35.099+0000 7fdc4b49a640 1 -- 172.21.15.155:0/1589008282 shutdown_connections 2024-03-25T18:30:35.101 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:35.099+0000 7fdc4b49a640 1 -- 172.21.15.155:0/1589008282 wait complete. 2024-03-25T18:30:35.101 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:35.099+0000 7fdc4b49a640 1 librados: shutdown 2024-03-25T18:30:35.111 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:41.919 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:42.037 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.036+0000 7f3c9a377640 10 monclient: get_monmap_and_config 2024-03-25T18:30:42.037 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.036+0000 7f3c9a377640 10 monclient: build_initial_monmap 2024-03-25T18:30:42.037 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.036+0000 7f3c9a377640 10 monclient: monmap: 2024-03-25T18:30:42.037 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:30:42.037 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:30:42.037 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:30:42.037155+0000 2024-03-25T18:30:42.038 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:30:42.037155+0000 2024-03-25T18:30:42.038 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:42.038 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:30:42.038 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:30:42.038 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:30:42.038 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:30:42.038 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:30:42.039 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.037+0000 7f3c9a377640 1 Processor -- start 2024-03-25T18:30:42.039 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.037+0000 7f3c9a377640 1 -- start start 2024-03-25T18:30:42.039 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.037+0000 7f3c9a377640 10 monclient: init 2024-03-25T18:30:42.040 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.037+0000 7f3c9a377640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:42.040 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.037+0000 7f3c9a377640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:30:42.040 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.038+0000 7f3c9a377640 10 monclient(hunting): picked mon.noname-a con 0x7f3c940a98f0 addr v1:172.21.15.155:6789/0 2024-03-25T18:30:42.040 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.038+0000 7f3c9a377640 10 monclient(hunting): picked mon.noname-b con 0x7f3c9413f710 addr v1:172.21.15.155:6790/0 2024-03-25T18:30:42.040 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.038+0000 7f3c9a377640 10 monclient(hunting): picked mon.noname-c con 0x7f3c941430c0 addr v1:172.21.15.167:6789/0 2024-03-25T18:30:42.040 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.038+0000 7f3c9a377640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c94142520 con 0x7f3c940a98f0 2024-03-25T18:30:42.040 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.038+0000 7f3c9a377640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c940b28a0 con 0x7f3c941430c0 2024-03-25T18:30:42.040 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.038+0000 7f3c9a377640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c940b2a30 con 0x7f3c9413f710 2024-03-25T18:30:42.040 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.038+0000 7f3c9a377640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:42.040 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.038+0000 7f3c9a377640 10 monclient(hunting): authenticate will time out at 7888.347815s 2024-03-25T18:30:42.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.048+0000 7f3c937fe640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f3c9413f710 0x7f3c941429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:51742/0 (socket says 172.21.15.155:51742) 2024-03-25T18:30:42.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.048+0000 7f3c937fe640 1 -- 172.21.15.155:0/2209248708 learned_addr learned my addr 172.21.15.155:0/2209248708 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:30:42.068 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.066+0000 7f3c92ffd640 1 -- 172.21.15.155:0/2209248708 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3472960094 0 0) 0x7f3c940b2a30 con 0x7f3c9413f710 2024-03-25T18:30:42.068 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.066+0000 7f3c92ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:42.068 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.066+0000 7f3c92ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:42.068 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.066+0000 7f3c92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:42.068 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.066+0000 7f3c92ffd640 10 monclient(hunting): my global_id is 4496 2024-03-25T18:30:42.068 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.066+0000 7f3c92ffd640 1 -- 172.21.15.155:0/2209248708 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c7c003520 con 0x7f3c9413f710 2024-03-25T18:30:42.074 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.073+0000 7f3c92ffd640 1 -- 172.21.15.155:0/2209248708 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4164739072 0 0) 0x7f3c94142520 con 0x7f3c940a98f0 2024-03-25T18:30:42.074 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.073+0000 7f3c92ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:42.075 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.073+0000 7f3c92ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:42.075 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.073+0000 7f3c92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:42.075 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.073+0000 7f3c92ffd640 10 monclient(hunting): my global_id is 4488 2024-03-25T18:30:42.075 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.073+0000 7f3c92ffd640 1 -- 172.21.15.155:0/2209248708 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c940b2a30 con 0x7f3c940a98f0 2024-03-25T18:30:42.079 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.076+0000 7f3c92ffd640 1 -- 172.21.15.155:0/2209248708 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3976496927 0 0) 0x7f3c940b28a0 con 0x7f3c941430c0 2024-03-25T18:30:42.079 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.076+0000 7f3c92ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:42.079 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.076+0000 7f3c92ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:42.079 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.076+0000 7f3c92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:42.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.076+0000 7f3c92ffd640 10 monclient(hunting): my global_id is 4483 2024-03-25T18:30:42.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.076+0000 7f3c92ffd640 1 -- 172.21.15.155:0/2209248708 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c94142520 con 0x7f3c941430c0 2024-03-25T18:30:42.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.076+0000 7f3c92ffd640 1 -- 172.21.15.155:0/2209248708 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3142527001 0 0) 0x7f3c7c003520 con 0x7f3c9413f710 2024-03-25T18:30:42.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.076+0000 7f3c92ffd640 1 -- 172.21.15.155:0/2209248708 >> v1:172.21.15.167:6789/0 conn(0x7f3c941430c0 legacy=0x7f3c9414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:42.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.076+0000 7f3c92ffd640 1 -- 172.21.15.155:0/2209248708 >> v1:172.21.15.155:6789/0 conn(0x7f3c940a98f0 legacy=0x7f3c940a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:42.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.076+0000 7f3c92ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:30:42.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.076+0000 7f3c92ffd640 1 monclient: found mon.noname-b 2024-03-25T18:30:42.081 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.076+0000 7f3c92ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:42.081 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.076+0000 7f3c92ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-25T18:30:42.081 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.076+0000 7f3c92ffd640 1 -- 172.21.15.155:0/2209248708 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c940b44c0 con 0x7f3c9413f710 2024-03-25T18:30:42.081 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.076+0000 7f3c92ffd640 10 monclient: _finish_auth 0 2024-03-25T18:30:42.082 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.076+0000 7f3c92ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:30:42.082 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.076+0000 7f3c92ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:42.082 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.076+0000 7f3c92ffd640 1 -- 172.21.15.155:0/2209248708 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f3c88001690 con 0x7f3c9413f710 2024-03-25T18:30:42.082 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.076+0000 7f3c92ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:42.082 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.076+0000 7f3c92ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:30:42.082 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.076+0000 7f3c92ffd640 10 monclient: dump: 2024-03-25T18:30:42.083 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:30:42.083 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:30:42.083 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:42.083 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:42.083 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:42.083 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:30:42.083 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:30:42.083 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:30:42.084 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:30:42.084 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:30:42.084 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.076+0000 7f3c9a377640 5 monclient: authenticate success, global_id 4496 2024-03-25T18:30:42.084 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.076+0000 7f3c9a377640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:42.088 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.081+0000 7f3c92ffd640 1 -- 172.21.15.155:0/2209248708 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c880017f0 con 0x7f3c9413f710 2024-03-25T18:30:42.088 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.081+0000 7f3c92ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:42.088 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.081+0000 7f3c92ffd640 1 -- 172.21.15.155:0/2209248708 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f3c88001c00 con 0x7f3c9413f710 2024-03-25T18:30:42.088 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.081+0000 7f3c92ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:42.088 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.081+0000 7f3c92ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:30:42.088 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.081+0000 7f3c92ffd640 10 monclient: dump: 2024-03-25T18:30:42.089 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:30:42.089 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:30:42.089 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:42.089 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:42.089 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:42.089 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:30:42.089 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:30:42.089 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:30:42.090 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:30:42.090 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:30:42.090 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.081+0000 7f3c9a377640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:42.090 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.081+0000 7f3c9a377640 4 set_mon_vals no callback set 2024-03-25T18:30:42.090 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.082+0000 7f3c9a377640 10 monclient: shutdown 2024-03-25T18:30:42.090 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.082+0000 7f3c9a377640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:42.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.082+0000 7f3c9a377640 1 -- 172.21.15.155:0/2209248708 >> v1:172.21.15.155:6790/0 conn(0x7f3c9413f710 legacy=0x7f3c941429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:42.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.082+0000 7f3c9a377640 1 -- 172.21.15.155:0/2209248708 shutdown_connections 2024-03-25T18:30:42.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.082+0000 7f3c9a377640 1 -- 172.21.15.155:0/2209248708 >> 172.21.15.155:0/2209248708 conn(0x7f3c940b7ea0 msgr2=0x7f3c940b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:42.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.082+0000 7f3c9a377640 1 -- 172.21.15.155:0/2209248708 shutdown_connections 2024-03-25T18:30:42.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.082+0000 7f3c9a377640 1 -- 172.21.15.155:0/2209248708 wait complete. 2024-03-25T18:30:42.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.083+0000 7f3c9a377640 10 monclient: build_initial_monmap 2024-03-25T18:30:42.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.083+0000 7f3c9a377640 10 monclient: monmap: 2024-03-25T18:30:42.092 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:30:42.092 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:42.092 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:30:42.092 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:30:42.092 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:42.092 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:30:42.092 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:30:42.093 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:30:42.093 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:30:42.093 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:30:42.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.084+0000 7f3c9a377640 1 librados: starting msgr at 2024-03-25T18:30:42.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.084+0000 7f3c9a377640 1 librados: starting objecter 2024-03-25T18:30:42.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.084+0000 7f3c9a377640 1 Processor -- start 2024-03-25T18:30:42.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.084+0000 7f3c9a377640 1 -- start start 2024-03-25T18:30:42.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.084+0000 7f3c9a377640 1 librados: setting wanted keys 2024-03-25T18:30:42.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.084+0000 7f3c9a377640 1 librados: calling monclient init 2024-03-25T18:30:42.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.084+0000 7f3c9a377640 10 monclient: init 2024-03-25T18:30:42.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.085+0000 7f3c9a377640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:42.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.085+0000 7f3c9a377640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:30:42.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.085+0000 7f3c9a377640 10 monclient(hunting): picked mon.noname-c con 0x7f3c940a98f0 addr v1:172.21.15.155:6790/0 2024-03-25T18:30:42.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.085+0000 7f3c9a377640 10 monclient(hunting): picked mon.noname-a con 0x7f3c9413f710 addr v1:172.21.15.155:6789/0 2024-03-25T18:30:42.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.085+0000 7f3c9a377640 10 monclient(hunting): picked mon.noname-b con 0x7f3c941430c0 addr v1:172.21.15.167:6789/0 2024-03-25T18:30:42.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.085+0000 7f3c9a377640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c940b1870 con 0x7f3c9413f710 2024-03-25T18:30:42.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.085+0000 7f3c9a377640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c940ac910 con 0x7f3c941430c0 2024-03-25T18:30:42.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.085+0000 7f3c9a377640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c940abb00 con 0x7f3c940a98f0 2024-03-25T18:30:42.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.085+0000 7f3c9a377640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:42.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.085+0000 7f3c9a377640 10 monclient(hunting): authenticate will time out at 7888.394830s 2024-03-25T18:30:42.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.093+0000 7f3c937fe640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f3c9413f710 0x7f3c940ac200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:44556/0 (socket says 172.21.15.155:44556) 2024-03-25T18:30:42.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.093+0000 7f3c937fe640 1 -- 172.21.15.155:0/3329641002 learned_addr learned my addr 172.21.15.155:0/3329641002 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:30:42.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.094+0000 7f3c90ff9640 10 client.?.objecter ms_handle_connect 0x7f3c9413f710 2024-03-25T18:30:42.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.094+0000 7f3c90ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:42.099 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.098+0000 7f3c90ff9640 10 client.?.objecter ms_handle_connect 0x7f3c940a98f0 2024-03-25T18:30:42.100 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.098+0000 7f3c90ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:42.100 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.099+0000 7f3c90ff9640 10 client.?.objecter ms_handle_connect 0x7f3c941430c0 2024-03-25T18:30:42.100 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.099+0000 7f3c90ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:42.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.102+0000 7f3c90ff9640 1 -- 172.21.15.155:0/3329641002 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 134587250 0 0) 0x7f3c940b44c0 con 0x7f3c9413f710 2024-03-25T18:30:42.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.102+0000 7f3c90ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:30:42.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.102+0000 7f3c90ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:42.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.102+0000 7f3c90ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:42.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.102+0000 7f3c90ff9640 10 monclient(hunting): my global_id is 4491 2024-03-25T18:30:42.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.102+0000 7f3c90ff9640 1 -- 172.21.15.155:0/3329641002 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c80004eb0 con 0x7f3c9413f710 2024-03-25T18:30:42.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.106+0000 7f3c90ff9640 1 -- 172.21.15.155:0/3329641002 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 523609613 0 0) 0x7f3c940ac910 con 0x7f3c941430c0 2024-03-25T18:30:42.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.106+0000 7f3c90ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:30:42.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.106+0000 7f3c90ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:42.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.106+0000 7f3c90ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:42.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.106+0000 7f3c90ff9640 10 monclient(hunting): my global_id is 4486 2024-03-25T18:30:42.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.106+0000 7f3c90ff9640 1 -- 172.21.15.155:0/3329641002 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c940b44c0 con 0x7f3c941430c0 2024-03-25T18:30:42.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.120+0000 7f3c90ff9640 1 -- 172.21.15.155:0/3329641002 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 57131184 0 0) 0x7f3c80004eb0 con 0x7f3c9413f710 2024-03-25T18:30:42.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.120+0000 7f3c90ff9640 1 -- 172.21.15.155:0/3329641002 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c940ac910 con 0x7f3c9413f710 2024-03-25T18:30:42.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.120+0000 7f3c90ff9640 1 -- 172.21.15.155:0/3329641002 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f3c88001a20 con 0x7f3c9413f710 2024-03-25T18:30:42.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.120+0000 7f3c90ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:42.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.120+0000 7f3c90ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:30:42.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.120+0000 7f3c90ff9640 10 monclient(hunting): dump: 2024-03-25T18:30:42.122 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:30:42.122 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:30:42.122 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:42.122 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:42.122 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:42.122 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:30:42.122 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:30:42.122 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:30:42.122 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:30:42.122 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:30:42.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.120+0000 7f3c90ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:42.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.121+0000 7f3c90ff9640 1 -- 172.21.15.155:0/3329641002 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1177327094 0 0) 0x7f3c940abb00 con 0x7f3c940a98f0 2024-03-25T18:30:42.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.121+0000 7f3c90ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:30:42.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.121+0000 7f3c90ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:42.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.121+0000 7f3c90ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:42.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.121+0000 7f3c90ff9640 10 monclient(hunting): my global_id is 4499 2024-03-25T18:30:42.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.121+0000 7f3c90ff9640 1 -- 172.21.15.155:0/3329641002 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c80004eb0 con 0x7f3c940a98f0 2024-03-25T18:30:42.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.134+0000 7f3c90ff9640 1 -- 172.21.15.155:0/3329641002 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2333119220 0 0) 0x7f3c940b44c0 con 0x7f3c941430c0 2024-03-25T18:30:42.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.134+0000 7f3c90ff9640 1 -- 172.21.15.155:0/3329641002 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c940abb00 con 0x7f3c941430c0 2024-03-25T18:30:42.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.134+0000 7f3c90ff9640 1 -- 172.21.15.155:0/3329641002 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f3c8c001b50 con 0x7f3c941430c0 2024-03-25T18:30:42.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.134+0000 7f3c90ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:42.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.134+0000 7f3c90ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:42.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.134+0000 7f3c90ff9640 10 monclient(hunting): dump: 2024-03-25T18:30:42.136 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:30:42.136 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:30:42.136 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:42.136 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:42.137 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:42.137 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:30:42.137 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:30:42.137 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:30:42.137 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:30:42.137 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:30:42.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.137+0000 7f3c90ff9640 1 -- 172.21.15.155:0/3329641002 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 114173297 0 0) 0x7f3c940ac910 con 0x7f3c9413f710 2024-03-25T18:30:42.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.137+0000 7f3c90ff9640 1 -- 172.21.15.155:0/3329641002 >> v1:172.21.15.155:6790/0 conn(0x7f3c940a98f0 legacy=0x7f3c94138670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:42.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.137+0000 7f3c90ff9640 1 -- 172.21.15.155:0/3329641002 >> v1:172.21.15.167:6789/0 conn(0x7f3c941430c0 legacy=0x7f3c941e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:42.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.137+0000 7f3c90ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:30:42.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.137+0000 7f3c90ff9640 1 monclient: found mon.a 2024-03-25T18:30:42.141 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.137+0000 7f3c90ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:42.141 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.137+0000 7f3c90ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:30:42.141 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.137+0000 7f3c90ff9640 1 -- 172.21.15.155:0/3329641002 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c940abde0 con 0x7f3c9413f710 2024-03-25T18:30:42.141 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.137+0000 7f3c90ff9640 10 monclient: _finish_auth 0 2024-03-25T18:30:42.141 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.137+0000 7f3c90ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:30:42.141 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.137+0000 7f3c90ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:42.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.137+0000 7f3c9a377640 5 monclient: authenticate success, global_id 4491 2024-03-25T18:30:42.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.137+0000 7f3c9a377640 10 monclient: _renew_subs 2024-03-25T18:30:42.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.137+0000 7f3c9a377640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:30:42.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.137+0000 7f3c9a377640 1 -- 172.21.15.155:0/3329641002 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c94138f40 con 0x7f3c9413f710 2024-03-25T18:30:42.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.137+0000 7f3c9a377640 10 client.4491.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:42.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.137+0000 7f3c9a377640 10 monclient: _renew_subs 2024-03-25T18:30:42.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.137+0000 7f3c9a377640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:30:42.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.137+0000 7f3c9a377640 1 -- 172.21.15.155:0/3329641002 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c941363f0 con 0x7f3c9413f710 2024-03-25T18:30:42.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.137+0000 7f3c9a377640 1 librados: init done 2024-03-25T18:30:42.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.138+0000 7f3c9a377640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:42.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.138+0000 7f3c9a377640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:42.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.138+0000 7f3c9a377640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:30:42.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.138+0000 7f3c9a377640 1 -- 172.21.15.155:0/3329641002 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c940627b0 con 0x7f3c9413f710 2024-03-25T18:30:42.430 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.429+0000 7f3c90ff9640 1 -- 172.21.15.155:0/3329641002 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c88001c60 con 0x7f3c9413f710 2024-03-25T18:30:42.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.429+0000 7f3c90ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:42.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.429+0000 7f3c90ff9640 1 -- 172.21.15.155:0/3329641002 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f3c88004c30 con 0x7f3c9413f710 2024-03-25T18:30:42.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.429+0000 7f3c90ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:42.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.429+0000 7f3c90ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:30:42.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.429+0000 7f3c90ff9640 10 monclient: dump: 2024-03-25T18:30:42.431 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:30:42.431 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:30:42.431 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:42.431 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:42.431 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:42.431 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:30:42.431 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:30:42.431 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:30:42.431 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:30:42.432 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:30:42.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.429+0000 7f3c927fc640 4 set_mon_vals no callback set 2024-03-25T18:30:42.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.430+0000 7f3c90ff9640 1 -- 172.21.15.155:0/3329641002 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7f3c8801ac00 con 0x7f3c9413f710 2024-03-25T18:30:42.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.431+0000 7f3c93fff640 10 monclient: get_auth_request con 0x7f3c8006df20 auth_method 0 2024-03-25T18:30:42.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.431+0000 7f3c90ff9640 10 client.4491.objecter ms_handle_connect 0x7f3c8006df20 2024-03-25T18:30:42.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.441+0000 7f3c90ff9640 1 -- 172.21.15.155:0/3329641002 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7f3c88054860 con 0x7f3c9413f710 2024-03-25T18:30:42.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.441+0000 7f3c90ff9640 10 client.4491.objecter ms_dispatch 0x7f3c94000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:30:42.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.441+0000 7f3c90ff9640 3 client.4491.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:30:42.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.441+0000 7f3c90ff9640 3 client.4491.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:30:42.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.441+0000 7f3c90ff9640 20 client.4491.objecter dump_active .. 0 homeless 2024-03-25T18:30:42.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.441+0000 7f3c90ff9640 1 -- 172.21.15.155:0/3329641002 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3c880501a0 con 0x7f3c9413f710 2024-03-25T18:30:42.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.441+0000 7f3c90ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:42.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.441+0000 7f3c90ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:42.689 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.687+0000 7f3c9a377640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:42.689 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.687+0000 7f3c9a377640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:42.689 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.687+0000 7f3c9a377640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:30:42.689 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.687+0000 7f3c9a377640 1 -- 172.21.15.155:0/3329641002 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3c941434a0 con 0x7f3c9413f710 2024-03-25T18:30:42.718 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.715+0000 7f3c90ff9640 1 -- 172.21.15.155:0/3329641002 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7f3c880522a0 con 0x7f3c9413f710 2024-03-25T18:30:42.718 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.715+0000 7f3c90ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:42.718 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.715+0000 7f3c90ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:42.718 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:30:42.719 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:42.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.717+0000 7f3c9a377640 10 librados: watch_flush enter 2024-03-25T18:30:42.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.717+0000 7f3c9a377640 10 librados: watch_flush exit 2024-03-25T18:30:42.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.717+0000 7f3c9a377640 20 client.4491.objecter shutdown clearing up homeless session... 2024-03-25T18:30:42.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.717+0000 7f3c9a377640 10 client.4491.objecter successfully canceled tick 2024-03-25T18:30:42.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.718+0000 7f3c9a377640 1 -- 172.21.15.155:0/3329641002 >> v1:172.21.15.167:6800/2919070503 conn(0x7f3c8006df20 legacy=0x7f3c800703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:42.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.718+0000 7f3c9a377640 10 monclient: shutdown 2024-03-25T18:30:42.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.718+0000 7f3c9a377640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:42.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.718+0000 7f3c9a377640 1 -- 172.21.15.155:0/3329641002 >> v1:172.21.15.155:6789/0 conn(0x7f3c9413f710 legacy=0x7f3c940ac200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:42.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.718+0000 7f3c9a377640 1 -- 172.21.15.155:0/3329641002 shutdown_connections 2024-03-25T18:30:42.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.718+0000 7f3c9a377640 1 -- 172.21.15.155:0/3329641002 >> 172.21.15.155:0/3329641002 conn(0x7f3c940b7ea0 msgr2=0x7f3c94074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:42.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.718+0000 7f3c9a377640 1 -- 172.21.15.155:0/3329641002 shutdown_connections 2024-03-25T18:30:42.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.718+0000 7f3c9a377640 1 -- 172.21.15.155:0/3329641002 wait complete. 2024-03-25T18:30:42.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:42.718+0000 7f3c9a377640 1 librados: shutdown 2024-03-25T18:30:42.730 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:46.533 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:06:20.137 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:46.541 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:06:20.145 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:46.567 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:06:20.171 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:46.590 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:06:20.169 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:46.595 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:06:20.173 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:46.612 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:06:20.217 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:46.622 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:06:20.227 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:46.636 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:06:20.215 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:46.657 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:06:20.262 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:49.537 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:49.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.659+0000 7f6f75b48640 10 monclient: get_monmap_and_config 2024-03-25T18:30:49.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.659+0000 7f6f75b48640 10 monclient: build_initial_monmap 2024-03-25T18:30:49.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.659+0000 7f6f75b48640 10 monclient: monmap: 2024-03-25T18:30:49.661 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:30:49.661 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:30:49.661 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:30:49.660522+0000 2024-03-25T18:30:49.661 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:30:49.660522+0000 2024-03-25T18:30:49.661 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:49.661 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:30:49.661 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:30:49.661 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:30:49.661 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:30:49.661 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:30:49.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.660+0000 7f6f75b48640 1 Processor -- start 2024-03-25T18:30:49.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.660+0000 7f6f75b48640 1 -- start start 2024-03-25T18:30:49.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.660+0000 7f6f75b48640 10 monclient: init 2024-03-25T18:30:49.663 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.661+0000 7f6f75b48640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:49.663 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.661+0000 7f6f75b48640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:30:49.663 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.661+0000 7f6f75b48640 10 monclient(hunting): picked mon.noname-a con 0x7f6f700a98f0 addr v1:172.21.15.155:6789/0 2024-03-25T18:30:49.663 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.661+0000 7f6f75b48640 10 monclient(hunting): picked mon.noname-c con 0x7f6f7013f710 addr v1:172.21.15.167:6789/0 2024-03-25T18:30:49.663 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.661+0000 7f6f75b48640 10 monclient(hunting): picked mon.noname-b con 0x7f6f701430c0 addr v1:172.21.15.155:6790/0 2024-03-25T18:30:49.663 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.661+0000 7f6f75b48640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f70142520 con 0x7f6f700a98f0 2024-03-25T18:30:49.663 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.661+0000 7f6f75b48640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f700b28a0 con 0x7f6f7013f710 2024-03-25T18:30:49.663 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.661+0000 7f6f75b48640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f700b2a30 con 0x7f6f701430c0 2024-03-25T18:30:49.663 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.661+0000 7f6f75b48640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:49.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.661+0000 7f6f75b48640 10 monclient(hunting): authenticate will time out at 7895.971380s 2024-03-25T18:30:49.670 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.668+0000 7f6f6effd640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f6f7013f710 0x7f6f701429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:34760/0 (socket says 172.21.15.155:34760) 2024-03-25T18:30:49.670 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.669+0000 7f6f6effd640 1 -- 172.21.15.155:0/3619322747 learned_addr learned my addr 172.21.15.155:0/3619322747 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:30:49.680 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.679+0000 7f6f6e7fc640 1 -- 172.21.15.155:0/3619322747 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2146225965 0 0) 0x7f6f70142520 con 0x7f6f700a98f0 2024-03-25T18:30:49.680 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.679+0000 7f6f6e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:30:49.681 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.679+0000 7f6f6e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:49.681 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.679+0000 7f6f6e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:49.681 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.679+0000 7f6f6e7fc640 10 monclient(hunting): my global_id is 4494 2024-03-25T18:30:49.681 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.679+0000 7f6f6e7fc640 1 -- 172.21.15.155:0/3619322747 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f58003520 con 0x7f6f700a98f0 2024-03-25T18:30:49.683 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.682+0000 7f6f6e7fc640 1 -- 172.21.15.155:0/3619322747 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3853810847 0 0) 0x7f6f700b28a0 con 0x7f6f7013f710 2024-03-25T18:30:49.683 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.682+0000 7f6f6e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:30:49.683 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.682+0000 7f6f6e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:49.683 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.682+0000 7f6f6e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:49.683 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.682+0000 7f6f6e7fc640 10 monclient(hunting): my global_id is 4489 2024-03-25T18:30:49.684 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.682+0000 7f6f6e7fc640 1 -- 172.21.15.155:0/3619322747 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f70142520 con 0x7f6f7013f710 2024-03-25T18:30:49.691 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.690+0000 7f6f6e7fc640 1 -- 172.21.15.155:0/3619322747 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1933055496 0 0) 0x7f6f700b2a30 con 0x7f6f701430c0 2024-03-25T18:30:49.691 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.690+0000 7f6f6e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:30:49.691 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.690+0000 7f6f6e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:49.691 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.690+0000 7f6f6e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:49.691 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.690+0000 7f6f6e7fc640 10 monclient(hunting): my global_id is 4502 2024-03-25T18:30:49.692 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.690+0000 7f6f6e7fc640 1 -- 172.21.15.155:0/3619322747 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f700b28a0 con 0x7f6f701430c0 2024-03-25T18:30:49.707 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.705+0000 7f6f6e7fc640 1 -- 172.21.15.155:0/3619322747 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3753572016 0 0) 0x7f6f58003520 con 0x7f6f700a98f0 2024-03-25T18:30:49.707 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.705+0000 7f6f6e7fc640 1 -- 172.21.15.155:0/3619322747 >> v1:172.21.15.155:6790/0 conn(0x7f6f701430c0 legacy=0x7f6f7014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:49.707 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.705+0000 7f6f6e7fc640 1 -- 172.21.15.155:0/3619322747 >> v1:172.21.15.167:6789/0 conn(0x7f6f7013f710 legacy=0x7f6f701429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:49.707 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.706+0000 7f6f6e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:30:49.707 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.706+0000 7f6f6e7fc640 1 monclient: found mon.noname-a 2024-03-25T18:30:49.707 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.706+0000 7f6f6e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:49.707 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.706+0000 7f6f6e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-25T18:30:49.707 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.706+0000 7f6f6e7fc640 1 -- 172.21.15.155:0/3619322747 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f700b44c0 con 0x7f6f700a98f0 2024-03-25T18:30:49.707 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.706+0000 7f6f6e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:30:49.707 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.706+0000 7f6f6e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:30:49.708 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.706+0000 7f6f6e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:49.708 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.706+0000 7f6f6e7fc640 1 -- 172.21.15.155:0/3619322747 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f6f60001690 con 0x7f6f700a98f0 2024-03-25T18:30:49.708 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.706+0000 7f6f75b48640 5 monclient: authenticate success, global_id 4494 2024-03-25T18:30:49.708 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.706+0000 7f6f75b48640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:49.708 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.706+0000 7f6f6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:49.708 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.706+0000 7f6f6e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:30:49.708 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.706+0000 7f6f6e7fc640 10 monclient: dump: 2024-03-25T18:30:49.708 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:30:49.708 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:30:49.708 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:49.708 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:49.708 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:49.708 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:30:49.708 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:30:49.708 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:30:49.708 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:30:49.709 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:30:49.709 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.706+0000 7f6f75b48640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:49.727 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.725+0000 7f6f6e7fc640 1 -- 172.21.15.155:0/3619322747 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6f600017f0 con 0x7f6f700a98f0 2024-03-25T18:30:49.727 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.725+0000 7f6f6e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:49.727 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.725+0000 7f6f6e7fc640 1 -- 172.21.15.155:0/3619322747 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f6f60001c00 con 0x7f6f700a98f0 2024-03-25T18:30:49.727 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.725+0000 7f6f6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:49.727 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.725+0000 7f6f6e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:30:49.727 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.725+0000 7f6f6e7fc640 10 monclient: dump: 2024-03-25T18:30:49.727 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:30:49.727 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:30:49.727 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:49.727 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:49.727 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:49.728 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:30:49.728 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:30:49.728 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:30:49.728 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:30:49.728 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:30:49.728 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.725+0000 7f6f75b48640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:49.728 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.725+0000 7f6f75b48640 4 set_mon_vals no callback set 2024-03-25T18:30:49.729 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.726+0000 7f6f75b48640 10 monclient: shutdown 2024-03-25T18:30:49.729 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.726+0000 7f6f75b48640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:49.729 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.726+0000 7f6f75b48640 1 -- 172.21.15.155:0/3619322747 >> v1:172.21.15.155:6789/0 conn(0x7f6f700a98f0 legacy=0x7f6f700a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:49.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.726+0000 7f6f75b48640 1 -- 172.21.15.155:0/3619322747 shutdown_connections 2024-03-25T18:30:49.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.726+0000 7f6f75b48640 1 -- 172.21.15.155:0/3619322747 >> 172.21.15.155:0/3619322747 conn(0x7f6f700b7ea0 msgr2=0x7f6f700b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:49.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.726+0000 7f6f75b48640 1 -- 172.21.15.155:0/3619322747 shutdown_connections 2024-03-25T18:30:49.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.726+0000 7f6f75b48640 1 -- 172.21.15.155:0/3619322747 wait complete. 2024-03-25T18:30:49.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.727+0000 7f6f75b48640 10 monclient: build_initial_monmap 2024-03-25T18:30:49.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.727+0000 7f6f75b48640 10 monclient: monmap: 2024-03-25T18:30:49.730 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:30:49.730 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:49.730 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:30:49.730 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:30:49.730 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:49.730 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:30:49.730 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:30:49.730 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:30:49.731 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:30:49.731 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:30:49.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.728+0000 7f6f75b48640 1 librados: starting msgr at 2024-03-25T18:30:49.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.728+0000 7f6f75b48640 1 librados: starting objecter 2024-03-25T18:30:49.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.728+0000 7f6f75b48640 1 Processor -- start 2024-03-25T18:30:49.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.728+0000 7f6f75b48640 1 -- start start 2024-03-25T18:30:49.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.728+0000 7f6f75b48640 1 librados: setting wanted keys 2024-03-25T18:30:49.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.728+0000 7f6f75b48640 1 librados: calling monclient init 2024-03-25T18:30:49.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.728+0000 7f6f75b48640 10 monclient: init 2024-03-25T18:30:49.732 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.729+0000 7f6f75b48640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:49.732 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.729+0000 7f6f75b48640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:30:49.732 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.729+0000 7f6f75b48640 10 monclient(hunting): picked mon.noname-a con 0x7f6f700a98f0 addr v1:172.21.15.155:6789/0 2024-03-25T18:30:49.732 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.729+0000 7f6f75b48640 10 monclient(hunting): picked mon.noname-c con 0x7f6f7013f710 addr v1:172.21.15.155:6790/0 2024-03-25T18:30:49.732 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.729+0000 7f6f75b48640 10 monclient(hunting): picked mon.noname-b con 0x7f6f701430c0 addr v1:172.21.15.167:6789/0 2024-03-25T18:30:49.732 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.729+0000 7f6f75b48640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f701db2f0 con 0x7f6f700a98f0 2024-03-25T18:30:49.732 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.729+0000 7f6f75b48640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f70139180 con 0x7f6f701430c0 2024-03-25T18:30:49.732 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.729+0000 7f6f75b48640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f70137670 con 0x7f6f7013f710 2024-03-25T18:30:49.732 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.729+0000 7f6f75b48640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:49.732 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.729+0000 7f6f75b48640 10 monclient(hunting): authenticate will time out at 7896.039482s 2024-03-25T18:30:49.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.742+0000 7f6f6f7fe640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f6f700a98f0 0x7f6f701dabe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:43156/0 (socket says 172.21.15.155:43156) 2024-03-25T18:30:49.743 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.742+0000 7f6f6f7fe640 1 -- 172.21.15.155:0/4001915432 learned_addr learned my addr 172.21.15.155:0/4001915432 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:30:49.746 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.745+0000 7f6f57fff640 10 client.?.objecter ms_handle_connect 0x7f6f700a98f0 2024-03-25T18:30:49.746 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.745+0000 7f6f57fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:49.748 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.747+0000 7f6f57fff640 10 client.?.objecter ms_handle_connect 0x7f6f701430c0 2024-03-25T18:30:49.748 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.747+0000 7f6f57fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:49.754 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.753+0000 7f6f57fff640 10 client.?.objecter ms_handle_connect 0x7f6f7013f710 2024-03-25T18:30:49.754 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.753+0000 7f6f57fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:49.761 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.760+0000 7f6f57fff640 1 -- 172.21.15.155:0/4001915432 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1098635059 0 0) 0x7f6f70139180 con 0x7f6f701430c0 2024-03-25T18:30:49.761 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.760+0000 7f6f57fff640 10 monclient(hunting): _negotiate 2024-03-25T18:30:49.761 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.760+0000 7f6f57fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:49.761 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.760+0000 7f6f57fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:49.761 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.760+0000 7f6f57fff640 10 monclient(hunting): my global_id is 4492 2024-03-25T18:30:49.761 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.760+0000 7f6f57fff640 1 -- 172.21.15.155:0/4001915432 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f5c004eb0 con 0x7f6f701430c0 2024-03-25T18:30:49.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.765+0000 7f6f57fff640 1 -- 172.21.15.155:0/4001915432 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1036784434 0 0) 0x7f6f70137670 con 0x7f6f7013f710 2024-03-25T18:30:49.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.765+0000 7f6f57fff640 10 monclient(hunting): _negotiate 2024-03-25T18:30:49.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.765+0000 7f6f57fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:49.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.765+0000 7f6f57fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:49.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.765+0000 7f6f57fff640 10 monclient(hunting): my global_id is 4505 2024-03-25T18:30:49.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.765+0000 7f6f57fff640 1 -- 172.21.15.155:0/4001915432 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f70139180 con 0x7f6f7013f710 2024-03-25T18:30:49.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.768+0000 7f6f57fff640 1 -- 172.21.15.155:0/4001915432 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 125017917 0 0) 0x7f6f701db2f0 con 0x7f6f700a98f0 2024-03-25T18:30:49.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.768+0000 7f6f57fff640 10 monclient(hunting): _negotiate 2024-03-25T18:30:49.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.768+0000 7f6f57fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:49.770 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.768+0000 7f6f57fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:49.770 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.768+0000 7f6f57fff640 10 monclient(hunting): my global_id is 4497 2024-03-25T18:30:49.770 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.768+0000 7f6f57fff640 1 -- 172.21.15.155:0/4001915432 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f70137670 con 0x7f6f700a98f0 2024-03-25T18:30:49.781 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.780+0000 7f6f57fff640 1 -- 172.21.15.155:0/4001915432 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2240039144 0 0) 0x7f6f70137670 con 0x7f6f700a98f0 2024-03-25T18:30:49.781 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.780+0000 7f6f57fff640 1 -- 172.21.15.155:0/4001915432 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6f701db2f0 con 0x7f6f700a98f0 2024-03-25T18:30:49.781 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.780+0000 7f6f57fff640 1 -- 172.21.15.155:0/4001915432 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f6f60001b00 con 0x7f6f700a98f0 2024-03-25T18:30:49.781 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.780+0000 7f6f57fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:49.781 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.780+0000 7f6f57fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:30:49.781 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.780+0000 7f6f57fff640 10 monclient(hunting): dump: 2024-03-25T18:30:49.781 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:30:49.781 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:30:49.782 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:49.782 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:49.782 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:49.782 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:30:49.782 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:30:49.782 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:30:49.782 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:30:49.782 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:30:49.782 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.780+0000 7f6f57fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:49.792 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.791+0000 7f6f57fff640 1 -- 172.21.15.155:0/4001915432 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4189597688 0 0) 0x7f6f70139180 con 0x7f6f7013f710 2024-03-25T18:30:49.792 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.791+0000 7f6f57fff640 1 -- 172.21.15.155:0/4001915432 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6f70137670 con 0x7f6f7013f710 2024-03-25T18:30:49.792 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.791+0000 7f6f57fff640 1 -- 172.21.15.155:0/4001915432 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f6f64001930 con 0x7f6f7013f710 2024-03-25T18:30:49.792 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.791+0000 7f6f57fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:49.792 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.791+0000 7f6f57fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:30:49.792 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.791+0000 7f6f57fff640 10 monclient(hunting): dump: 2024-03-25T18:30:49.792 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:30:49.792 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:30:49.793 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:49.793 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:49.793 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:49.793 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:30:49.793 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:30:49.793 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:30:49.793 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:30:49.793 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:30:49.793 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.791+0000 7f6f57fff640 1 -- 172.21.15.155:0/4001915432 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 154447666 0 0) 0x7f6f5c004eb0 con 0x7f6f701430c0 2024-03-25T18:30:49.793 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.791+0000 7f6f57fff640 1 -- 172.21.15.155:0/4001915432 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6f70139180 con 0x7f6f701430c0 2024-03-25T18:30:49.793 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.791+0000 7f6f57fff640 1 -- 172.21.15.155:0/4001915432 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f6f68001bd0 con 0x7f6f701430c0 2024-03-25T18:30:49.793 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.791+0000 7f6f57fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:49.793 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.791+0000 7f6f57fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:49.793 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.791+0000 7f6f57fff640 10 monclient(hunting): dump: 2024-03-25T18:30:49.793 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:30:49.793 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:30:49.794 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:49.794 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:49.794 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:49.794 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:30:49.794 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:30:49.794 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:30:49.794 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:30:49.794 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:30:49.803 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.802+0000 7f6f57fff640 1 -- 172.21.15.155:0/4001915432 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3968896709 0 0) 0x7f6f701db2f0 con 0x7f6f700a98f0 2024-03-25T18:30:49.804 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.802+0000 7f6f57fff640 1 -- 172.21.15.155:0/4001915432 >> v1:172.21.15.155:6790/0 conn(0x7f6f7013f710 legacy=0x7f6f70132f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:49.804 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.802+0000 7f6f57fff640 1 -- 172.21.15.155:0/4001915432 >> v1:172.21.15.167:6789/0 conn(0x7f6f701430c0 legacy=0x7f6f701e2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:49.804 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.802+0000 7f6f57fff640 10 monclient: _finish_hunting 0 2024-03-25T18:30:49.804 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.802+0000 7f6f57fff640 1 monclient: found mon.a 2024-03-25T18:30:49.804 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.802+0000 7f6f57fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:49.804 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.802+0000 7f6f57fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:30:49.804 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.802+0000 7f6f57fff640 1 -- 172.21.15.155:0/4001915432 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f70137200 con 0x7f6f700a98f0 2024-03-25T18:30:49.804 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.802+0000 7f6f57fff640 10 monclient: _finish_auth 0 2024-03-25T18:30:49.804 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.802+0000 7f6f57fff640 10 monclient: _check_auth_tickets 2024-03-25T18:30:49.804 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.802+0000 7f6f57fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:49.804 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.802+0000 7f6f75b48640 5 monclient: authenticate success, global_id 4497 2024-03-25T18:30:49.804 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.802+0000 7f6f75b48640 10 monclient: _renew_subs 2024-03-25T18:30:49.804 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.802+0000 7f6f75b48640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:30:49.805 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.802+0000 7f6f75b48640 1 -- 172.21.15.155:0/4001915432 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f700b7290 con 0x7f6f700a98f0 2024-03-25T18:30:49.805 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.802+0000 7f6f75b48640 10 client.4497.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:49.805 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.802+0000 7f6f75b48640 10 monclient: _renew_subs 2024-03-25T18:30:49.805 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.802+0000 7f6f75b48640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:30:49.805 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.802+0000 7f6f75b48640 1 -- 172.21.15.155:0/4001915432 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f70133680 con 0x7f6f700a98f0 2024-03-25T18:30:49.805 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.802+0000 7f6f75b48640 1 librados: init done 2024-03-25T18:30:49.805 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.803+0000 7f6f75b48640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:49.805 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.803+0000 7f6f75b48640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:49.806 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.803+0000 7f6f75b48640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:30:49.806 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:49.803+0000 7f6f75b48640 1 -- 172.21.15.155:0/4001915432 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f70137800 con 0x7f6f700a98f0 2024-03-25T18:30:50.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:50.135+0000 7f6f57fff640 1 -- 172.21.15.155:0/4001915432 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6f60001230 con 0x7f6f700a98f0 2024-03-25T18:30:50.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:50.135+0000 7f6f57fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:50.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:50.135+0000 7f6f57fff640 1 -- 172.21.15.155:0/4001915432 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f6f60003a40 con 0x7f6f700a98f0 2024-03-25T18:30:50.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:50.135+0000 7f6f57fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:50.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:50.135+0000 7f6f57fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:30:50.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:50.135+0000 7f6f57fff640 10 monclient: dump: 2024-03-25T18:30:50.137 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:30:50.137 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:30:50.138 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:50.138 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:50.138 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:50.138 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:30:50.138 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:30:50.138 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:30:50.138 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:30:50.138 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:30:50.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:50.135+0000 7f6f6dffb640 4 set_mon_vals no callback set 2024-03-25T18:30:50.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:50.136+0000 7f6f57fff640 1 -- 172.21.15.155:0/4001915432 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7f6f60003c60 con 0x7f6f700a98f0 2024-03-25T18:30:50.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:50.137+0000 7f6f6effd640 10 monclient: get_auth_request con 0x7f6f5c06e0b0 auth_method 0 2024-03-25T18:30:50.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:50.137+0000 7f6f57fff640 10 client.4497.objecter ms_handle_connect 0x7f6f5c06e0b0 2024-03-25T18:30:50.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:50.141+0000 7f6f57fff640 1 -- 172.21.15.155:0/4001915432 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7f6f60054820 con 0x7f6f700a98f0 2024-03-25T18:30:50.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:50.142+0000 7f6f57fff640 10 client.4497.objecter ms_dispatch 0x7f6f70000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:30:50.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:50.142+0000 7f6f57fff640 3 client.4497.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:30:50.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:50.142+0000 7f6f57fff640 3 client.4497.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:30:50.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:50.142+0000 7f6f57fff640 20 client.4497.objecter dump_active .. 0 homeless 2024-03-25T18:30:50.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:50.142+0000 7f6f57fff640 1 -- 172.21.15.155:0/4001915432 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6f60050330 con 0x7f6f700a98f0 2024-03-25T18:30:50.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:50.142+0000 7f6f57fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:50.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:50.142+0000 7f6f57fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:50.382 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:50.381+0000 7f6f75b48640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:50.382 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:50.381+0000 7f6f75b48640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:50.383 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:50.381+0000 7f6f75b48640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:30:50.383 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:50.381+0000 7f6f75b48640 1 -- 172.21.15.155:0/4001915432 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6f700b7420 con 0x7f6f700a98f0 2024-03-25T18:30:50.434 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:50.432+0000 7f6f57fff640 1 -- 172.21.15.155:0/4001915432 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7f6f60052260 con 0x7f6f700a98f0 2024-03-25T18:30:50.434 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:50.433+0000 7f6f57fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:50.434 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:50.433+0000 7f6f57fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:50.434 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:30:50.434 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:50.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:50.435+0000 7f6f75b48640 10 librados: watch_flush enter 2024-03-25T18:30:50.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:50.435+0000 7f6f75b48640 10 librados: watch_flush exit 2024-03-25T18:30:50.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:50.435+0000 7f6f75b48640 20 client.4497.objecter shutdown clearing up homeless session... 2024-03-25T18:30:50.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:50.435+0000 7f6f75b48640 10 client.4497.objecter successfully canceled tick 2024-03-25T18:30:50.438 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:50.436+0000 7f6f75b48640 1 -- 172.21.15.155:0/4001915432 >> v1:172.21.15.167:6800/2919070503 conn(0x7f6f5c06e0b0 legacy=0x7f6f5c070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:50.438 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:50.436+0000 7f6f75b48640 10 monclient: shutdown 2024-03-25T18:30:50.438 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:50.436+0000 7f6f75b48640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:50.438 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:50.436+0000 7f6f75b48640 1 -- 172.21.15.155:0/4001915432 >> v1:172.21.15.155:6789/0 conn(0x7f6f700a98f0 legacy=0x7f6f701dabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:50.438 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:50.436+0000 7f6f75b48640 1 -- 172.21.15.155:0/4001915432 shutdown_connections 2024-03-25T18:30:50.438 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:50.436+0000 7f6f75b48640 1 -- 172.21.15.155:0/4001915432 >> 172.21.15.155:0/4001915432 conn(0x7f6f700b7ea0 msgr2=0x7f6f70074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:50.438 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:50.436+0000 7f6f75b48640 1 -- 172.21.15.155:0/4001915432 shutdown_connections 2024-03-25T18:30:50.438 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:50.436+0000 7f6f75b48640 1 -- 172.21.15.155:0/4001915432 wait complete. 2024-03-25T18:30:50.438 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:50.436+0000 7f6f75b48640 1 librados: shutdown 2024-03-25T18:30:50.448 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:57.254 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:57.371 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.370+0000 7f3850a4a640 10 monclient: get_monmap_and_config 2024-03-25T18:30:57.372 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.370+0000 7f3850a4a640 10 monclient: build_initial_monmap 2024-03-25T18:30:57.372 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.370+0000 7f3850a4a640 10 monclient: monmap: 2024-03-25T18:30:57.372 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:30:57.372 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:30:57.372 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:30:57.371722+0000 2024-03-25T18:30:57.372 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:30:57.371722+0000 2024-03-25T18:30:57.372 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:57.372 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:30:57.372 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:30:57.372 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:30:57.372 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:30:57.372 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:30:57.374 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.371+0000 7f3850a4a640 1 Processor -- start 2024-03-25T18:30:57.374 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.371+0000 7f3850a4a640 1 -- start start 2024-03-25T18:30:57.374 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.371+0000 7f3850a4a640 10 monclient: init 2024-03-25T18:30:57.374 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.372+0000 7f3850a4a640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:57.374 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.372+0000 7f3850a4a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:30:57.374 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.372+0000 7f3850a4a640 10 monclient(hunting): picked mon.noname-a con 0x7f384c06ab80 addr v1:172.21.15.155:6789/0 2024-03-25T18:30:57.374 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.372+0000 7f3850a4a640 10 monclient(hunting): picked mon.noname-b con 0x7f384c06af60 addr v1:172.21.15.155:6790/0 2024-03-25T18:30:57.375 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.372+0000 7f3850a4a640 10 monclient(hunting): picked mon.noname-c con 0x7f384c0b00b0 addr v1:172.21.15.167:6789/0 2024-03-25T18:30:57.375 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.372+0000 7f3850a4a640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f384c0b0500 con 0x7f384c06ab80 2024-03-25T18:30:57.375 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.372+0000 7f3850a4a640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f384c13ffa0 con 0x7f384c0b00b0 2024-03-25T18:30:57.375 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.372+0000 7f3850a4a640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f384c0af520 con 0x7f384c06af60 2024-03-25T18:30:57.375 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.372+0000 7f3850a4a640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:57.375 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.372+0000 7f3850a4a640 10 monclient(hunting): authenticate will time out at 7903.682150s 2024-03-25T18:30:57.380 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.379+0000 7f3849d9b640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f384c06af60 0x7f384c13f890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:45410/0 (socket says 172.21.15.155:45410) 2024-03-25T18:30:57.380 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.379+0000 7f3849d9b640 1 -- 172.21.15.155:0/494502108 learned_addr learned my addr 172.21.15.155:0/494502108 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:30:57.395 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.393+0000 7f384959a640 1 -- 172.21.15.155:0/494502108 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2193011497 0 0) 0x7f384c0af520 con 0x7f384c06af60 2024-03-25T18:30:57.395 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.393+0000 7f384959a640 10 monclient(hunting): _negotiate 2024-03-25T18:30:57.395 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.393+0000 7f384959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:57.395 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.393+0000 7f384959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:57.395 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.393+0000 7f384959a640 10 monclient(hunting): my global_id is 4508 2024-03-25T18:30:57.395 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.393+0000 7f384959a640 1 -- 172.21.15.155:0/494502108 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3834003520 con 0x7f384c06af60 2024-03-25T18:30:57.407 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.405+0000 7f384959a640 1 -- 172.21.15.155:0/494502108 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2691204917 0 0) 0x7f3834003520 con 0x7f384c06af60 2024-03-25T18:30:57.407 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.405+0000 7f384959a640 1 -- 172.21.15.155:0/494502108 >> v1:172.21.15.167:6789/0 conn(0x7f384c0b00b0 legacy=0x7f384c14dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:57.407 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.405+0000 7f384959a640 1 -- 172.21.15.155:0/494502108 >> v1:172.21.15.155:6789/0 conn(0x7f384c06ab80 legacy=0x7f384c0ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:57.407 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.405+0000 7f384959a640 10 monclient: _finish_hunting 0 2024-03-25T18:30:57.407 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.405+0000 7f384959a640 1 monclient: found mon.noname-b 2024-03-25T18:30:57.407 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.405+0000 7f384959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:57.407 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.405+0000 7f384959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-25T18:30:57.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.405+0000 7f384959a640 1 -- 172.21.15.155:0/494502108 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f384c072890 con 0x7f384c06af60 2024-03-25T18:30:57.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.405+0000 7f384959a640 10 monclient: _finish_auth 0 2024-03-25T18:30:57.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.405+0000 7f384959a640 10 monclient: _check_auth_tickets 2024-03-25T18:30:57.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.405+0000 7f384959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:57.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.405+0000 7f384959a640 1 -- 172.21.15.155:0/494502108 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f3840001690 con 0x7f384c06af60 2024-03-25T18:30:57.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.405+0000 7f3850a4a640 5 monclient: authenticate success, global_id 4508 2024-03-25T18:30:57.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.405+0000 7f3850a4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:57.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.405+0000 7f384959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:57.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.405+0000 7f384959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:30:57.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.405+0000 7f384959a640 10 monclient: dump: 2024-03-25T18:30:57.408 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:30:57.408 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:30:57.408 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:57.409 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:57.409 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:57.409 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:30:57.409 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:30:57.409 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:30:57.409 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:30:57.409 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:30:57.409 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.405+0000 7f3850a4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:57.415 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.413+0000 7f384959a640 1 -- 172.21.15.155:0/494502108 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f38400017f0 con 0x7f384c06af60 2024-03-25T18:30:57.415 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.413+0000 7f384959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:57.415 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.413+0000 7f384959a640 1 -- 172.21.15.155:0/494502108 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f3840001c00 con 0x7f384c06af60 2024-03-25T18:30:57.415 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.413+0000 7f3850a4a640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:57.415 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.413+0000 7f3850a4a640 4 set_mon_vals no callback set 2024-03-25T18:30:57.415 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.413+0000 7f384959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:57.415 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.413+0000 7f384959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:30:57.415 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.413+0000 7f384959a640 10 monclient: dump: 2024-03-25T18:30:57.415 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:30:57.415 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:30:57.415 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:57.415 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:57.416 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:57.416 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:30:57.416 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:30:57.416 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:30:57.416 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:30:57.416 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:30:57.416 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.413+0000 7f3850a4a640 10 monclient: shutdown 2024-03-25T18:30:57.416 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.413+0000 7f3850a4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:57.416 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.413+0000 7f3850a4a640 1 -- 172.21.15.155:0/494502108 >> v1:172.21.15.155:6790/0 conn(0x7f384c06af60 legacy=0x7f384c13f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:57.416 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.414+0000 7f3850a4a640 1 -- 172.21.15.155:0/494502108 shutdown_connections 2024-03-25T18:30:57.416 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.414+0000 7f3850a4a640 1 -- 172.21.15.155:0/494502108 >> 172.21.15.155:0/494502108 conn(0x7f384c0b4a80 msgr2=0x7f384c0b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:57.416 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.414+0000 7f3850a4a640 1 -- 172.21.15.155:0/494502108 shutdown_connections 2024-03-25T18:30:57.416 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.414+0000 7f3850a4a640 1 -- 172.21.15.155:0/494502108 wait complete. 2024-03-25T18:30:57.418 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.415+0000 7f3850a4a640 10 monclient: build_initial_monmap 2024-03-25T18:30:57.418 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.415+0000 7f3850a4a640 10 monclient: monmap: 2024-03-25T18:30:57.418 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:30:57.418 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:57.418 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:30:57.418 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:30:57.418 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:57.419 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:30:57.419 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:30:57.419 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:30:57.419 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:30:57.419 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:30:57.419 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.415+0000 7f3850a4a640 1 librados: starting msgr at 2024-03-25T18:30:57.419 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.415+0000 7f3850a4a640 1 librados: starting objecter 2024-03-25T18:30:57.419 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.415+0000 7f3850a4a640 1 Processor -- start 2024-03-25T18:30:57.419 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.415+0000 7f3850a4a640 1 -- start start 2024-03-25T18:30:57.419 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.415+0000 7f3850a4a640 1 librados: setting wanted keys 2024-03-25T18:30:57.419 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.415+0000 7f3850a4a640 1 librados: calling monclient init 2024-03-25T18:30:57.419 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.415+0000 7f3850a4a640 10 monclient: init 2024-03-25T18:30:57.419 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.416+0000 7f3850a4a640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:57.419 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.416+0000 7f3850a4a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:30:57.419 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.416+0000 7f3850a4a640 10 monclient(hunting): picked mon.noname-c con 0x7f384c06ab80 addr v1:172.21.15.155:6790/0 2024-03-25T18:30:57.419 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.416+0000 7f3850a4a640 10 monclient(hunting): picked mon.noname-a con 0x7f384c0b00b0 addr v1:172.21.15.155:6789/0 2024-03-25T18:30:57.420 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.416+0000 7f3850a4a640 10 monclient(hunting): picked mon.noname-b con 0x7f384c143720 addr v1:172.21.15.167:6789/0 2024-03-25T18:30:57.420 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.416+0000 7f3850a4a640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f384c11cfe0 con 0x7f384c0b00b0 2024-03-25T18:30:57.420 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.416+0000 7f3850a4a640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f384c11e040 con 0x7f384c143720 2024-03-25T18:30:57.420 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.416+0000 7f3850a4a640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f384c1dac50 con 0x7f384c06ab80 2024-03-25T18:30:57.420 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.416+0000 7f3850a4a640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:57.420 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.416+0000 7f3850a4a640 10 monclient(hunting): authenticate will time out at 7903.726211s 2024-03-25T18:30:57.421 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.420+0000 7f384ad9d640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f384c143720 0x7f384c13ac30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:48682/0 (socket says 172.21.15.155:48682) 2024-03-25T18:30:57.421 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.420+0000 7f384ad9d640 1 -- 172.21.15.155:0/756399294 learned_addr learned my addr 172.21.15.155:0/756399294 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:30:57.422 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.421+0000 7f383affd640 10 client.?.objecter ms_handle_connect 0x7f384c143720 2024-03-25T18:30:57.422 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.421+0000 7f383affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:57.426 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.425+0000 7f383affd640 1 -- 172.21.15.155:0/756399294 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1412629474 0 0) 0x7f384c11e040 con 0x7f384c143720 2024-03-25T18:30:57.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.425+0000 7f383affd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:57.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.425+0000 7f383affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:57.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.425+0000 7f383affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:57.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.425+0000 7f383affd640 10 monclient(hunting): my global_id is 4498 2024-03-25T18:30:57.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.425+0000 7f383affd640 1 -- 172.21.15.155:0/756399294 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f382c004eb0 con 0x7f384c143720 2024-03-25T18:30:57.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.430+0000 7f383affd640 10 client.?.objecter ms_handle_connect 0x7f384c06ab80 2024-03-25T18:30:57.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.430+0000 7f383affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:57.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.432+0000 7f383affd640 10 client.?.objecter ms_handle_connect 0x7f384c0b00b0 2024-03-25T18:30:57.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.432+0000 7f383affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:57.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.436+0000 7f383affd640 1 -- 172.21.15.155:0/756399294 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4070892428 0 0) 0x7f382c004eb0 con 0x7f384c143720 2024-03-25T18:30:57.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.436+0000 7f383affd640 1 -- 172.21.15.155:0/756399294 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f384c11e040 con 0x7f384c143720 2024-03-25T18:30:57.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.436+0000 7f383affd640 1 -- 172.21.15.155:0/756399294 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f3844001c30 con 0x7f384c143720 2024-03-25T18:30:57.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.436+0000 7f383affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:57.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.436+0000 7f383affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:30:57.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.436+0000 7f383affd640 10 monclient(hunting): dump: 2024-03-25T18:30:57.437 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:30:57.438 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:30:57.438 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:57.438 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:57.438 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:57.438 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:30:57.438 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:30:57.438 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:30:57.438 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:30:57.438 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:30:57.438 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.436+0000 7f383affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:57.443 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.442+0000 7f383affd640 1 -- 172.21.15.155:0/756399294 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3548153373 0 0) 0x7f384c11cfe0 con 0x7f384c0b00b0 2024-03-25T18:30:57.443 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.442+0000 7f383affd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:57.443 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.442+0000 7f383affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:57.443 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.442+0000 7f383affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:57.443 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.442+0000 7f383affd640 10 monclient(hunting): my global_id is 4503 2024-03-25T18:30:57.444 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.442+0000 7f383affd640 1 -- 172.21.15.155:0/756399294 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f382c004eb0 con 0x7f384c0b00b0 2024-03-25T18:30:57.444 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.443+0000 7f383affd640 1 -- 172.21.15.155:0/756399294 <== mon.1 v1:172.21.15.167:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2040736322 0 0) 0x7f384c11e040 con 0x7f384c143720 2024-03-25T18:30:57.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.443+0000 7f383affd640 1 -- 172.21.15.155:0/756399294 >> v1:172.21.15.155:6790/0 conn(0x7f384c06ab80 legacy=0x7f384c11c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:57.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.443+0000 7f383affd640 1 -- 172.21.15.155:0/756399294 >> v1:172.21.15.155:6789/0 conn(0x7f384c0b00b0 legacy=0x7f384c1db170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:57.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.443+0000 7f383affd640 10 monclient: _finish_hunting 0 2024-03-25T18:30:57.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.443+0000 7f383affd640 1 monclient: found mon.b 2024-03-25T18:30:57.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.443+0000 7f383affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:57.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.443+0000 7f383affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:30:57.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.443+0000 7f383affd640 1 -- 172.21.15.155:0/756399294 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f384c1db880 con 0x7f384c143720 2024-03-25T18:30:57.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.443+0000 7f383affd640 10 monclient: _finish_auth 0 2024-03-25T18:30:57.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.443+0000 7f383affd640 10 monclient: _check_auth_tickets 2024-03-25T18:30:57.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.443+0000 7f383affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:57.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.443+0000 7f3850a4a640 5 monclient: authenticate success, global_id 4498 2024-03-25T18:30:57.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.443+0000 7f3850a4a640 10 monclient: _renew_subs 2024-03-25T18:30:57.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.443+0000 7f3850a4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:30:57.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.443+0000 7f3850a4a640 1 -- 172.21.15.155:0/756399294 --> v1:172.21.15.167:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f384c13b340 con 0x7f384c143720 2024-03-25T18:30:57.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.443+0000 7f3850a4a640 10 client.4498.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:57.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.443+0000 7f3850a4a640 10 monclient: _renew_subs 2024-03-25T18:30:57.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.443+0000 7f3850a4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:30:57.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.443+0000 7f3850a4a640 1 -- 172.21.15.155:0/756399294 --> v1:172.21.15.167:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f384c146240 con 0x7f384c143720 2024-03-25T18:30:57.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.443+0000 7f3850a4a640 1 librados: init done 2024-03-25T18:30:57.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.444+0000 7f3850a4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:57.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.444+0000 7f3850a4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:57.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.444+0000 7f3850a4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:30:57.447 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.444+0000 7f3850a4a640 1 -- 172.21.15.155:0/756399294 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f384c0627b0 con 0x7f384c143720 2024-03-25T18:30:57.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.717+0000 7f383affd640 1 -- 172.21.15.155:0/756399294 <== mon.1 v1:172.21.15.167:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3844003990 con 0x7f384c143720 2024-03-25T18:30:57.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.717+0000 7f383affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:57.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.717+0000 7f383affd640 1 -- 172.21.15.155:0/756399294 <== mon.1 v1:172.21.15.167:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f3844003de0 con 0x7f384c143720 2024-03-25T18:30:57.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.717+0000 7f383affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:57.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.717+0000 7f383affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:57.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.717+0000 7f383affd640 10 monclient: dump: 2024-03-25T18:30:57.719 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:30:57.719 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:30:57.719 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:57.720 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:30:57.720 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:57.720 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:30:57.720 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:30:57.720 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:30:57.720 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:30:57.720 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:30:57.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.717+0000 7f3848d99640 4 set_mon_vals no callback set 2024-03-25T18:30:57.721 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.718+0000 7f383affd640 1 -- 172.21.15.155:0/756399294 <== mon.1 v1:172.21.15.167:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7f3844004000 con 0x7f384c143720 2024-03-25T18:30:57.721 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.719+0000 7f384a59c640 10 monclient: get_auth_request con 0x7f382c06dd90 auth_method 0 2024-03-25T18:30:57.721 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.719+0000 7f383affd640 10 client.4498.objecter ms_handle_connect 0x7f382c06dd90 2024-03-25T18:30:57.725 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.724+0000 7f383affd640 1 -- 172.21.15.155:0/756399294 <== mon.1 v1:172.21.15.167:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7f3844054a20 con 0x7f384c143720 2024-03-25T18:30:57.725 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.724+0000 7f383affd640 10 client.4498.objecter ms_dispatch 0x7f384c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:30:57.725 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.724+0000 7f383affd640 3 client.4498.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:30:57.725 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.724+0000 7f383affd640 3 client.4498.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:30:57.725 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.724+0000 7f383affd640 20 client.4498.objecter dump_active .. 0 homeless 2024-03-25T18:30:57.726 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.725+0000 7f383affd640 1 -- 172.21.15.155:0/756399294 <== mon.1 v1:172.21.15.167:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3844050530 con 0x7f384c143720 2024-03-25T18:30:57.726 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.725+0000 7f383affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:57.726 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.725+0000 7f383affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:57.977 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.975+0000 7f3850a4a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:57.977 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.975+0000 7f3850a4a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:57.977 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.975+0000 7f3850a4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:30:57.977 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:57.975+0000 7f3850a4a640 1 -- 172.21.15.155:0/756399294 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f384c1dba10 con 0x7f384c143720 2024-03-25T18:30:58.002 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:58.001+0000 7f383affd640 1 -- 172.21.15.155:0/756399294 <== mon.1 v1:172.21.15.167:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7f3844052460 con 0x7f384c143720 2024-03-25T18:30:58.002 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:58.001+0000 7f383affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:58.003 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:58.001+0000 7f383affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:58.003 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:30:58.003 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:58.006 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:58.003+0000 7f3850a4a640 10 librados: watch_flush enter 2024-03-25T18:30:58.006 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:58.004+0000 7f3850a4a640 10 librados: watch_flush exit 2024-03-25T18:30:58.006 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:58.004+0000 7f3850a4a640 20 client.4498.objecter shutdown clearing up homeless session... 2024-03-25T18:30:58.006 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:58.004+0000 7f3850a4a640 10 client.4498.objecter successfully canceled tick 2024-03-25T18:30:58.006 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:58.004+0000 7f3850a4a640 1 -- 172.21.15.155:0/756399294 >> v1:172.21.15.167:6800/2919070503 conn(0x7f382c06dd90 legacy=0x7f382c070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:58.006 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:58.004+0000 7f3850a4a640 10 monclient: shutdown 2024-03-25T18:30:58.006 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:58.004+0000 7f3850a4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:58.006 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:58.004+0000 7f3850a4a640 1 -- 172.21.15.155:0/756399294 >> v1:172.21.15.167:6789/0 conn(0x7f384c143720 legacy=0x7f384c13ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:58.006 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:58.004+0000 7f3850a4a640 1 -- 172.21.15.155:0/756399294 shutdown_connections 2024-03-25T18:30:58.006 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:58.004+0000 7f3850a4a640 1 -- 172.21.15.155:0/756399294 >> 172.21.15.155:0/756399294 conn(0x7f384c0b4a80 msgr2=0x7f384c0b4e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:58.006 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:58.004+0000 7f3850a4a640 1 -- 172.21.15.155:0/756399294 shutdown_connections 2024-03-25T18:30:58.006 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:58.004+0000 7f3850a4a640 1 -- 172.21.15.155:0/756399294 wait complete. 2024-03-25T18:30:58.007 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:30:58.004+0000 7f3850a4a640 1 librados: shutdown 2024-03-25T18:30:58.018 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:31:04.825 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:31:04.940 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:04.938+0000 7f0ebd74d640 10 monclient: get_monmap_and_config 2024-03-25T18:31:04.940 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:04.938+0000 7f0ebd74d640 10 monclient: build_initial_monmap 2024-03-25T18:31:04.940 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:04.938+0000 7f0ebd74d640 10 monclient: monmap: 2024-03-25T18:31:04.940 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:31:04.940 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:31:04.940 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:31:04.940132+0000 2024-03-25T18:31:04.940 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:31:04.940132+0000 2024-03-25T18:31:04.940 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:04.941 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:31:04.941 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:31:04.941 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:31:04.941 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:31:04.941 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:31:04.942 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:04.938+0000 7f0ebd74d640 1 Processor -- start 2024-03-25T18:31:04.942 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:04.939+0000 7f0ebd74d640 1 -- start start 2024-03-25T18:31:04.942 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:04.939+0000 7f0ebd74d640 10 monclient: init 2024-03-25T18:31:04.942 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:04.939+0000 7f0ebd74d640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:04.942 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:04.939+0000 7f0ebd74d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:31:04.942 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:04.939+0000 7f0ebd74d640 10 monclient(hunting): picked mon.noname-a con 0x7f0eb806c380 addr v1:172.21.15.155:6789/0 2024-03-25T18:31:04.942 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:04.939+0000 7f0ebd74d640 10 monclient(hunting): picked mon.noname-c con 0x7f0eb80b12d0 addr v1:172.21.15.167:6789/0 2024-03-25T18:31:04.942 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:04.939+0000 7f0ebd74d640 10 monclient(hunting): picked mon.noname-b con 0x7f0eb80aedf0 addr v1:172.21.15.155:6790/0 2024-03-25T18:31:04.943 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:04.939+0000 7f0ebd74d640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eb80b16f0 con 0x7f0eb806c380 2024-03-25T18:31:04.943 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:04.939+0000 7f0ebd74d640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eb80adfe0 con 0x7f0eb80b12d0 2024-03-25T18:31:04.943 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:04.939+0000 7f0ebd74d640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eb80add00 con 0x7f0eb80aedf0 2024-03-25T18:31:04.943 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:04.939+0000 7f0ebd74d640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:04.943 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:04.939+0000 7f0ebd74d640 10 monclient(hunting): authenticate will time out at 7911.250596s 2024-03-25T18:31:04.951 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:04.949+0000 7f0eb67fc640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f0eb80b12d0 0x7f0eb80ae6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:53510/0 (socket says 172.21.15.155:53510) 2024-03-25T18:31:04.951 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:04.949+0000 7f0eb67fc640 1 -- 172.21.15.155:0/2520667388 learned_addr learned my addr 172.21.15.155:0/2520667388 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:31:04.963 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:04.961+0000 7f0eb5ffb640 1 -- 172.21.15.155:0/2520667388 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3550797465 0 0) 0x7f0eb80adfe0 con 0x7f0eb80b12d0 2024-03-25T18:31:04.963 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:04.961+0000 7f0eb5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:31:04.963 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:04.961+0000 7f0eb5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:04.963 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:04.961+0000 7f0eb5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:04.963 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:04.961+0000 7f0eb5ffb640 10 monclient(hunting): my global_id is 4501 2024-03-25T18:31:04.963 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:04.961+0000 7f0eb5ffb640 1 -- 172.21.15.155:0/2520667388 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ea0003520 con 0x7f0eb80b12d0 2024-03-25T18:31:04.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:04.982+0000 7f0eb5ffb640 1 -- 172.21.15.155:0/2520667388 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1664903262 0 0) 0x7f0eb80add00 con 0x7f0eb80aedf0 2024-03-25T18:31:04.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:04.982+0000 7f0eb5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:31:04.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:04.982+0000 7f0eb5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:04.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:04.982+0000 7f0eb5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:04.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:04.982+0000 7f0eb5ffb640 10 monclient(hunting): my global_id is 4514 2024-03-25T18:31:04.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:04.982+0000 7f0eb5ffb640 1 -- 172.21.15.155:0/2520667388 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0eb80adfe0 con 0x7f0eb80aedf0 2024-03-25T18:31:04.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:04.993+0000 7f0eb5ffb640 1 -- 172.21.15.155:0/2520667388 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1025881555 0 0) 0x7f0ea0003520 con 0x7f0eb80b12d0 2024-03-25T18:31:04.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:04.993+0000 7f0eb5ffb640 1 -- 172.21.15.155:0/2520667388 >> v1:172.21.15.155:6790/0 conn(0x7f0eb80aedf0 legacy=0x7f0eb814e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:04.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:04.993+0000 7f0eb5ffb640 1 -- 172.21.15.155:0/2520667388 >> v1:172.21.15.155:6789/0 conn(0x7f0eb806c380 legacy=0x7f0eb80b0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:04.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:04.993+0000 7f0eb5ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:31:04.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:04.993+0000 7f0eb5ffb640 1 monclient: found mon.noname-c 2024-03-25T18:31:04.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:04.993+0000 7f0eb5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:04.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:04.993+0000 7f0eb5ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.167:6789/0 2024-03-25T18:31:04.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:04.993+0000 7f0eb5ffb640 1 -- 172.21.15.155:0/2520667388 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eb80ae2c0 con 0x7f0eb80b12d0 2024-03-25T18:31:04.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:04.993+0000 7f0eb5ffb640 10 monclient: _finish_auth 0 2024-03-25T18:31:04.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:04.993+0000 7f0eb5ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:31:04.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:04.993+0000 7f0eb5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:04.997 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:04.993+0000 7f0eb5ffb640 1 -- 172.21.15.155:0/2520667388 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f0eac001690 con 0x7f0eb80b12d0 2024-03-25T18:31:04.997 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:04.993+0000 7f0ebd74d640 5 monclient: authenticate success, global_id 4501 2024-03-25T18:31:04.997 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:04.993+0000 7f0ebd74d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:04.997 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:04.993+0000 7f0eb5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:04.997 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:04.993+0000 7f0eb5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:31:04.997 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:04.993+0000 7f0eb5ffb640 10 monclient: dump: 2024-03-25T18:31:04.997 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:31:04.997 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:31:04.997 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:04.997 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:04.997 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:04.997 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:31:04.997 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:31:04.998 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:31:04.998 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:31:04.998 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:31:04.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:04.993+0000 7f0ebd74d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:05.012 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.010+0000 7f0eb5ffb640 1 -- 172.21.15.155:0/2520667388 <== mon.1 v1:172.21.15.167:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0eac0017f0 con 0x7f0eb80b12d0 2024-03-25T18:31:05.013 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.010+0000 7f0eb5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:05.013 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.010+0000 7f0eb5ffb640 1 -- 172.21.15.155:0/2520667388 <== mon.1 v1:172.21.15.167:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f0eac001b80 con 0x7f0eb80b12d0 2024-03-25T18:31:05.013 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.010+0000 7f0ebd74d640 10 monclient: get_monmap_and_config success 2024-03-25T18:31:05.013 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.010+0000 7f0ebd74d640 4 set_mon_vals no callback set 2024-03-25T18:31:05.013 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.010+0000 7f0eb5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:05.013 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.010+0000 7f0eb5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:31:05.013 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.010+0000 7f0eb5ffb640 10 monclient: dump: 2024-03-25T18:31:05.013 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:31:05.013 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:31:05.013 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:05.013 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:05.013 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:05.013 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:31:05.014 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:31:05.014 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:31:05.014 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:31:05.014 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:31:05.015 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.011+0000 7f0ebd74d640 10 monclient: shutdown 2024-03-25T18:31:05.015 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.011+0000 7f0ebd74d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:05.015 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.011+0000 7f0ebd74d640 1 -- 172.21.15.155:0/2520667388 >> v1:172.21.15.167:6789/0 conn(0x7f0eb80b12d0 legacy=0x7f0eb80ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:05.015 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.012+0000 7f0ebd74d640 1 -- 172.21.15.155:0/2520667388 shutdown_connections 2024-03-25T18:31:05.015 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.012+0000 7f0ebd74d640 1 -- 172.21.15.155:0/2520667388 >> 172.21.15.155:0/2520667388 conn(0x7f0eb80b5d10 msgr2=0x7f0eb80b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:05.015 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.012+0000 7f0ebd74d640 1 -- 172.21.15.155:0/2520667388 shutdown_connections 2024-03-25T18:31:05.015 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.012+0000 7f0ebd74d640 1 -- 172.21.15.155:0/2520667388 wait complete. 2024-03-25T18:31:05.016 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.013+0000 7f0ebd74d640 10 monclient: build_initial_monmap 2024-03-25T18:31:05.016 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.013+0000 7f0ebd74d640 10 monclient: monmap: 2024-03-25T18:31:05.016 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:31:05.016 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:31:05.016 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:31:05.016 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:31:05.016 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:05.016 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:31:05.016 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:31:05.016 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:31:05.017 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:31:05.017 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:31:05.017 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.013+0000 7f0ebd74d640 1 librados: starting msgr at 2024-03-25T18:31:05.017 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.013+0000 7f0ebd74d640 1 librados: starting objecter 2024-03-25T18:31:05.017 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.013+0000 7f0ebd74d640 1 Processor -- start 2024-03-25T18:31:05.017 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.014+0000 7f0ebd74d640 1 -- start start 2024-03-25T18:31:05.017 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.014+0000 7f0ebd74d640 1 librados: setting wanted keys 2024-03-25T18:31:05.017 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.014+0000 7f0ebd74d640 1 librados: calling monclient init 2024-03-25T18:31:05.017 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.014+0000 7f0ebd74d640 10 monclient: init 2024-03-25T18:31:05.018 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.014+0000 7f0ebd74d640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:05.018 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.014+0000 7f0ebd74d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:31:05.019 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.014+0000 7f0ebd74d640 10 monclient(hunting): picked mon.noname-b con 0x7f0eb806c380 addr v1:172.21.15.167:6789/0 2024-03-25T18:31:05.019 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.014+0000 7f0ebd74d640 10 monclient(hunting): picked mon.noname-c con 0x7f0eb80aedf0 addr v1:172.21.15.155:6790/0 2024-03-25T18:31:05.019 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.014+0000 7f0ebd74d640 10 monclient(hunting): picked mon.noname-a con 0x7f0eb80b12d0 addr v1:172.21.15.155:6789/0 2024-03-25T18:31:05.019 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.014+0000 7f0ebd74d640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eb806c8e0 con 0x7f0eb80b12d0 2024-03-25T18:31:05.019 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.014+0000 7f0ebd74d640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eb81dbb00 con 0x7f0eb806c380 2024-03-25T18:31:05.019 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.014+0000 7f0ebd74d640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eb81c4420 con 0x7f0eb80aedf0 2024-03-25T18:31:05.019 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.014+0000 7f0ebd74d640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:05.019 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.014+0000 7f0ebd74d640 10 monclient(hunting): authenticate will time out at 7911.325432s 2024-03-25T18:31:05.028 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.026+0000 7f0eb6ffd640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f0eb806c380 0x7f0eb81dabb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:53570/0 (socket says 172.21.15.155:53570) 2024-03-25T18:31:05.029 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.026+0000 7f0eb6ffd640 1 -- 172.21.15.155:0/1167197912 learned_addr learned my addr 172.21.15.155:0/1167197912 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:31:05.029 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.027+0000 7f0e9f7fe640 10 client.?.objecter ms_handle_connect 0x7f0eb806c380 2024-03-25T18:31:05.029 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.027+0000 7f0e9f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:05.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.032+0000 7f0e9f7fe640 1 -- 172.21.15.155:0/1167197912 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1246792705 0 0) 0x7f0eb81dbb00 con 0x7f0eb806c380 2024-03-25T18:31:05.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.032+0000 7f0e9f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:31:05.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.032+0000 7f0e9f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:05.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.032+0000 7f0e9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:05.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.032+0000 7f0e9f7fe640 10 monclient(hunting): my global_id is 4504 2024-03-25T18:31:05.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.032+0000 7f0e9f7fe640 1 -- 172.21.15.155:0/1167197912 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ea4004eb0 con 0x7f0eb806c380 2024-03-25T18:31:05.041 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.039+0000 7f0e9f7fe640 10 client.?.objecter ms_handle_connect 0x7f0eb80b12d0 2024-03-25T18:31:05.041 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.039+0000 7f0e9f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:05.044 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.042+0000 7f0e9f7fe640 1 -- 172.21.15.155:0/1167197912 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1016577866 0 0) 0x7f0ea4004eb0 con 0x7f0eb806c380 2024-03-25T18:31:05.044 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.042+0000 7f0e9f7fe640 1 -- 172.21.15.155:0/1167197912 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0eb81dbb00 con 0x7f0eb806c380 2024-03-25T18:31:05.044 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.042+0000 7f0e9f7fe640 1 -- 172.21.15.155:0/1167197912 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f0ea8001a30 con 0x7f0eb806c380 2024-03-25T18:31:05.045 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.042+0000 7f0e9f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:05.045 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.042+0000 7f0e9f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:31:05.045 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.042+0000 7f0e9f7fe640 10 monclient(hunting): dump: 2024-03-25T18:31:05.045 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:31:05.045 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:31:05.045 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:05.045 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:05.045 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:05.045 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:31:05.045 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:31:05.045 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:31:05.045 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:31:05.045 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:31:05.046 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.042+0000 7f0e9f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:31:05.047 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.045+0000 7f0e9f7fe640 10 client.?.objecter ms_handle_connect 0x7f0eb80aedf0 2024-03-25T18:31:05.047 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.045+0000 7f0e9f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:05.052 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.049+0000 7f0e9f7fe640 1 -- 172.21.15.155:0/1167197912 <== mon.1 v1:172.21.15.167:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3662958818 0 0) 0x7f0eb81dbb00 con 0x7f0eb806c380 2024-03-25T18:31:05.052 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.049+0000 7f0e9f7fe640 1 -- 172.21.15.155:0/1167197912 >> v1:172.21.15.155:6790/0 conn(0x7f0eb80aedf0 legacy=0x7f0eb81c31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:05.052 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.049+0000 7f0e9f7fe640 1 -- 172.21.15.155:0/1167197912 >> v1:172.21.15.155:6789/0 conn(0x7f0eb80b12d0 legacy=0x7f0eb81c07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:05.052 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.049+0000 7f0e9f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:31:05.052 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.049+0000 7f0e9f7fe640 1 monclient: found mon.b 2024-03-25T18:31:05.052 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.049+0000 7f0e9f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:05.052 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.049+0000 7f0e9f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:31:05.052 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.049+0000 7f0e9f7fe640 1 -- 172.21.15.155:0/1167197912 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eb81c4690 con 0x7f0eb806c380 2024-03-25T18:31:05.053 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.049+0000 7f0e9f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:31:05.053 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.049+0000 7f0e9f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:31:05.053 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.049+0000 7f0e9f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:05.053 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.049+0000 7f0ebd74d640 5 monclient: authenticate success, global_id 4504 2024-03-25T18:31:05.053 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.049+0000 7f0ebd74d640 10 monclient: _renew_subs 2024-03-25T18:31:05.053 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.049+0000 7f0ebd74d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:31:05.053 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.049+0000 7f0ebd74d640 1 -- 172.21.15.155:0/1167197912 --> v1:172.21.15.167:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0eb81c22a0 con 0x7f0eb806c380 2024-03-25T18:31:05.053 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.049+0000 7f0ebd74d640 10 client.4504.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:31:05.053 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.049+0000 7f0ebd74d640 10 monclient: _renew_subs 2024-03-25T18:31:05.053 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.049+0000 7f0ebd74d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:31:05.053 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.049+0000 7f0ebd74d640 1 -- 172.21.15.155:0/1167197912 --> v1:172.21.15.167:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0eb81c10d0 con 0x7f0eb806c380 2024-03-25T18:31:05.053 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.049+0000 7f0ebd74d640 1 librados: init done 2024-03-25T18:31:05.054 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.050+0000 7f0ebd74d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:05.054 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.050+0000 7f0ebd74d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:05.054 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.050+0000 7f0ebd74d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:31:05.054 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.050+0000 7f0ebd74d640 1 -- 172.21.15.155:0/1167197912 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0eb80627b0 con 0x7f0eb806c380 2024-03-25T18:31:05.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.273+0000 7f0e9f7fe640 1 -- 172.21.15.155:0/1167197912 <== mon.1 v1:172.21.15.167:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0ea8001cf0 con 0x7f0eb806c380 2024-03-25T18:31:05.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.273+0000 7f0e9f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:05.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.273+0000 7f0eb57fa640 4 set_mon_vals no callback set 2024-03-25T18:31:05.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.313+0000 7f0e9f7fe640 1 -- 172.21.15.155:0/1167197912 <== mon.1 v1:172.21.15.167:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f0ea8003a60 con 0x7f0eb806c380 2024-03-25T18:31:05.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.313+0000 7f0e9f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:05.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.313+0000 7f0e9f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:31:05.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.313+0000 7f0e9f7fe640 10 monclient: dump: 2024-03-25T18:31:05.315 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:31:05.315 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:31:05.315 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:05.315 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:05.316 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:05.316 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:31:05.316 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:31:05.316 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:31:05.316 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:31:05.316 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:31:05.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.321+0000 7f0e9f7fe640 1 -- 172.21.15.155:0/1167197912 <== mon.1 v1:172.21.15.167:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7f0ea8003c40 con 0x7f0eb806c380 2024-03-25T18:31:05.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.321+0000 7f0eb67fc640 10 monclient: get_auth_request con 0x7f0ea406dd20 auth_method 0 2024-03-25T18:31:05.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.321+0000 7f0e9f7fe640 10 client.4504.objecter ms_handle_connect 0x7f0ea406dd20 2024-03-25T18:31:05.327 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.325+0000 7f0e9f7fe640 1 -- 172.21.15.155:0/1167197912 <== mon.1 v1:172.21.15.167:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7f0ea80549c0 con 0x7f0eb806c380 2024-03-25T18:31:05.327 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.325+0000 7f0e9f7fe640 10 client.4504.objecter ms_dispatch 0x7f0eb8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:31:05.327 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.325+0000 7f0e9f7fe640 3 client.4504.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:31:05.327 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.325+0000 7f0e9f7fe640 3 client.4504.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:31:05.327 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.325+0000 7f0e9f7fe640 20 client.4504.objecter dump_active .. 0 homeless 2024-03-25T18:31:05.334 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.332+0000 7f0e9f7fe640 1 -- 172.21.15.155:0/1167197912 <== mon.1 v1:172.21.15.167:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0ea8050300 con 0x7f0eb806c380 2024-03-25T18:31:05.334 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.332+0000 7f0e9f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:05.334 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.332+0000 7f0e9f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:31:05.562 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.560+0000 7f0ebd74d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:05.563 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.560+0000 7f0ebd74d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:05.563 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.560+0000 7f0ebd74d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:31:05.563 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.560+0000 7f0ebd74d640 1 -- 172.21.15.155:0/1167197912 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0eb806d790 con 0x7f0eb806c380 2024-03-25T18:31:05.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.576+0000 7f0e9f7fe640 1 -- 172.21.15.155:0/1167197912 <== mon.1 v1:172.21.15.167:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7f0eb806d790 con 0x7f0eb806c380 2024-03-25T18:31:05.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.576+0000 7f0e9f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:05.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.576+0000 7f0e9f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:31:05.579 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:31:05.579 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:31:05.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.578+0000 7f0ebd74d640 10 librados: watch_flush enter 2024-03-25T18:31:05.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.578+0000 7f0ebd74d640 10 librados: watch_flush exit 2024-03-25T18:31:05.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.578+0000 7f0ebd74d640 20 client.4504.objecter shutdown clearing up homeless session... 2024-03-25T18:31:05.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.578+0000 7f0ebd74d640 10 client.4504.objecter successfully canceled tick 2024-03-25T18:31:05.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.578+0000 7f0ebd74d640 1 -- 172.21.15.155:0/1167197912 >> v1:172.21.15.167:6800/2919070503 conn(0x7f0ea406dd20 legacy=0x7f0ea40701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:05.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.578+0000 7f0ebd74d640 10 monclient: shutdown 2024-03-25T18:31:05.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.578+0000 7f0ebd74d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:05.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.578+0000 7f0ebd74d640 1 -- 172.21.15.155:0/1167197912 >> v1:172.21.15.167:6789/0 conn(0x7f0eb806c380 legacy=0x7f0eb81dabb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:05.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.578+0000 7f0ebd74d640 1 -- 172.21.15.155:0/1167197912 shutdown_connections 2024-03-25T18:31:05.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.578+0000 7f0ebd74d640 1 -- 172.21.15.155:0/1167197912 >> 172.21.15.155:0/1167197912 conn(0x7f0eb80b5d10 msgr2=0x7f0eb81cd600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:05.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.579+0000 7f0ebd74d640 1 -- 172.21.15.155:0/1167197912 shutdown_connections 2024-03-25T18:31:05.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.579+0000 7f0ebd74d640 1 -- 172.21.15.155:0/1167197912 wait complete. 2024-03-25T18:31:05.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:05.579+0000 7f0ebd74d640 1 librados: shutdown 2024-03-25T18:31:05.594 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:31:06.541 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:06:40.145 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:06.589 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:06:40.193 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:06.590 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:06:40.168 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:06.598 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:06:40.177 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:06.624 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:06:40.228 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:06.637 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:06:40.216 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:06.671 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:06:40.275 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:06.679 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:06:40.284 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:06.713 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:06:40.318 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:12.400 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:31:12.518 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.517+0000 7fb3a870e640 10 monclient: get_monmap_and_config 2024-03-25T18:31:12.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.517+0000 7fb3a870e640 10 monclient: build_initial_monmap 2024-03-25T18:31:12.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.517+0000 7fb3a870e640 10 monclient: monmap: 2024-03-25T18:31:12.519 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:31:12.519 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:31:12.519 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:31:12.518094+0000 2024-03-25T18:31:12.519 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:31:12.518094+0000 2024-03-25T18:31:12.519 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:12.519 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:31:12.519 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:31:12.519 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:31:12.519 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:31:12.519 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:31:12.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.517+0000 7fb3a870e640 1 Processor -- start 2024-03-25T18:31:12.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.517+0000 7fb3a870e640 1 -- start start 2024-03-25T18:31:12.520 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.517+0000 7fb3a870e640 10 monclient: init 2024-03-25T18:31:12.520 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.518+0000 7fb3a870e640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:12.520 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.518+0000 7fb3a870e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:31:12.521 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.518+0000 7fb3a870e640 10 monclient(hunting): picked mon.noname-a con 0x7fb3a00a98f0 addr v1:172.21.15.155:6789/0 2024-03-25T18:31:12.521 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.518+0000 7fb3a870e640 10 monclient(hunting): picked mon.noname-b con 0x7fb3a013f710 addr v1:172.21.15.155:6790/0 2024-03-25T18:31:12.521 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.518+0000 7fb3a870e640 10 monclient(hunting): picked mon.noname-c con 0x7fb3a01430c0 addr v1:172.21.15.167:6789/0 2024-03-25T18:31:12.521 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.518+0000 7fb3a870e640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3a0142520 con 0x7fb3a00a98f0 2024-03-25T18:31:12.521 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.518+0000 7fb3a870e640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3a00b28a0 con 0x7fb3a01430c0 2024-03-25T18:31:12.521 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.518+0000 7fb3a870e640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3a00b2a30 con 0x7fb3a013f710 2024-03-25T18:31:12.521 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.518+0000 7fb3a870e640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:12.521 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.518+0000 7fb3a870e640 10 monclient(hunting): authenticate will time out at 7918.828603s 2024-03-25T18:31:12.537 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.536+0000 7fb3a6cab640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7fb3a01430c0 0x7fb3a014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:35586/0 (socket says 172.21.15.155:35586) 2024-03-25T18:31:12.538 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.536+0000 7fb3a6cab640 1 -- 172.21.15.155:0/1044841686 learned_addr learned my addr 172.21.15.155:0/1044841686 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:31:12.550 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.549+0000 7fb3a54a8640 1 -- 172.21.15.155:0/1044841686 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2142330233 0 0) 0x7fb3a00b28a0 con 0x7fb3a01430c0 2024-03-25T18:31:12.550 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.549+0000 7fb3a54a8640 10 monclient(hunting): _negotiate 2024-03-25T18:31:12.551 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.549+0000 7fb3a54a8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:12.551 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.549+0000 7fb3a54a8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:12.551 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.549+0000 7fb3a54a8640 10 monclient(hunting): my global_id is 4507 2024-03-25T18:31:12.551 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.549+0000 7fb3a54a8640 1 -- 172.21.15.155:0/1044841686 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb38c003520 con 0x7fb3a01430c0 2024-03-25T18:31:12.557 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.555+0000 7fb3a54a8640 1 -- 172.21.15.155:0/1044841686 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3089780241 0 0) 0x7fb3a00b2a30 con 0x7fb3a013f710 2024-03-25T18:31:12.557 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.555+0000 7fb3a54a8640 10 monclient(hunting): _negotiate 2024-03-25T18:31:12.557 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.555+0000 7fb3a54a8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:12.557 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.555+0000 7fb3a54a8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:12.557 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.555+0000 7fb3a54a8640 10 monclient(hunting): my global_id is 4520 2024-03-25T18:31:12.557 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.555+0000 7fb3a54a8640 1 -- 172.21.15.155:0/1044841686 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3a00b28a0 con 0x7fb3a013f710 2024-03-25T18:31:12.560 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.558+0000 7fb3a54a8640 1 -- 172.21.15.155:0/1044841686 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2309255471 0 0) 0x7fb3a0142520 con 0x7fb3a00a98f0 2024-03-25T18:31:12.560 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.558+0000 7fb3a54a8640 10 monclient(hunting): _negotiate 2024-03-25T18:31:12.560 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.558+0000 7fb3a54a8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:12.561 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.558+0000 7fb3a54a8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:12.561 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.558+0000 7fb3a54a8640 10 monclient(hunting): my global_id is 4512 2024-03-25T18:31:12.561 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.558+0000 7fb3a54a8640 1 -- 172.21.15.155:0/1044841686 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3a00b2a30 con 0x7fb3a00a98f0 2024-03-25T18:31:12.561 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.559+0000 7fb3a54a8640 1 -- 172.21.15.155:0/1044841686 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3317506625 0 0) 0x7fb38c003520 con 0x7fb3a01430c0 2024-03-25T18:31:12.561 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.559+0000 7fb3a54a8640 1 -- 172.21.15.155:0/1044841686 >> v1:172.21.15.155:6790/0 conn(0x7fb3a013f710 legacy=0x7fb3a01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:12.561 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.559+0000 7fb3a54a8640 1 -- 172.21.15.155:0/1044841686 >> v1:172.21.15.155:6789/0 conn(0x7fb3a00a98f0 legacy=0x7fb3a00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:12.561 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.559+0000 7fb3a54a8640 10 monclient: _finish_hunting 0 2024-03-25T18:31:12.561 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.559+0000 7fb3a54a8640 1 monclient: found mon.noname-c 2024-03-25T18:31:12.561 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.559+0000 7fb3a54a8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:12.561 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.559+0000 7fb3a54a8640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.167:6789/0 2024-03-25T18:31:12.561 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.559+0000 7fb3a54a8640 1 -- 172.21.15.155:0/1044841686 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3a00b44c0 con 0x7fb3a01430c0 2024-03-25T18:31:12.561 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.559+0000 7fb3a54a8640 10 monclient: _finish_auth 0 2024-03-25T18:31:12.561 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.559+0000 7fb3a54a8640 10 monclient: _check_auth_tickets 2024-03-25T18:31:12.561 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.559+0000 7fb3a54a8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:12.561 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.559+0000 7fb3a54a8640 1 -- 172.21.15.155:0/1044841686 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fb398001a10 con 0x7fb3a01430c0 2024-03-25T18:31:12.562 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.559+0000 7fb3a870e640 5 monclient: authenticate success, global_id 4507 2024-03-25T18:31:12.562 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.559+0000 7fb3a870e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:12.562 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.559+0000 7fb3a54a8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:12.562 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.559+0000 7fb3a54a8640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:31:12.562 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.559+0000 7fb3a54a8640 10 monclient: dump: 2024-03-25T18:31:12.562 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:31:12.562 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:31:12.562 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:12.562 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:12.562 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:12.562 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:31:12.562 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:31:12.562 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:31:12.562 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:31:12.562 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:31:12.563 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.559+0000 7fb3a870e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:12.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.571+0000 7fb3a54a8640 1 -- 172.21.15.155:0/1044841686 <== mon.1 v1:172.21.15.167:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb398001bf0 con 0x7fb3a01430c0 2024-03-25T18:31:12.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.571+0000 7fb3a54a8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:12.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.571+0000 7fb3a54a8640 1 -- 172.21.15.155:0/1044841686 <== mon.1 v1:172.21.15.167:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fb398002000 con 0x7fb3a01430c0 2024-03-25T18:31:12.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.571+0000 7fb3a54a8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:12.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.571+0000 7fb3a54a8640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:31:12.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.571+0000 7fb3a54a8640 10 monclient: dump: 2024-03-25T18:31:12.573 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:31:12.573 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:31:12.573 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:12.573 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:12.573 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:12.573 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:31:12.573 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:31:12.573 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:31:12.573 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:31:12.574 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:31:12.574 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.571+0000 7fb3a870e640 10 monclient: get_monmap_and_config success 2024-03-25T18:31:12.574 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.571+0000 7fb3a870e640 4 set_mon_vals no callback set 2024-03-25T18:31:12.576 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.572+0000 7fb3a870e640 10 monclient: shutdown 2024-03-25T18:31:12.576 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.572+0000 7fb3a870e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:12.576 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.572+0000 7fb3a870e640 1 -- 172.21.15.155:0/1044841686 >> v1:172.21.15.167:6789/0 conn(0x7fb3a01430c0 legacy=0x7fb3a014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:12.576 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.572+0000 7fb3a870e640 1 -- 172.21.15.155:0/1044841686 shutdown_connections 2024-03-25T18:31:12.576 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.572+0000 7fb3a870e640 1 -- 172.21.15.155:0/1044841686 >> 172.21.15.155:0/1044841686 conn(0x7fb3a00b7ea0 msgr2=0x7fb3a00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:12.576 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.572+0000 7fb3a870e640 1 -- 172.21.15.155:0/1044841686 shutdown_connections 2024-03-25T18:31:12.576 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.572+0000 7fb3a870e640 1 -- 172.21.15.155:0/1044841686 wait complete. 2024-03-25T18:31:12.576 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.573+0000 7fb3a870e640 10 monclient: build_initial_monmap 2024-03-25T18:31:12.576 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.573+0000 7fb3a870e640 10 monclient: monmap: 2024-03-25T18:31:12.576 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:31:12.577 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:31:12.577 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:31:12.577 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:31:12.577 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:12.577 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:31:12.577 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:31:12.577 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:31:12.577 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:31:12.577 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:31:12.577 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.573+0000 7fb3a870e640 1 librados: starting msgr at 2024-03-25T18:31:12.577 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.573+0000 7fb3a870e640 1 librados: starting objecter 2024-03-25T18:31:12.577 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.574+0000 7fb3a870e640 1 Processor -- start 2024-03-25T18:31:12.577 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.574+0000 7fb3a870e640 1 -- start start 2024-03-25T18:31:12.577 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.574+0000 7fb3a870e640 1 librados: setting wanted keys 2024-03-25T18:31:12.577 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.574+0000 7fb3a870e640 1 librados: calling monclient init 2024-03-25T18:31:12.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.574+0000 7fb3a870e640 10 monclient: init 2024-03-25T18:31:12.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.574+0000 7fb3a870e640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:12.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.574+0000 7fb3a870e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:31:12.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.574+0000 7fb3a870e640 10 monclient(hunting): picked mon.noname-b con 0x7fb3a00a98f0 addr v1:172.21.15.167:6789/0 2024-03-25T18:31:12.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.574+0000 7fb3a870e640 10 monclient(hunting): picked mon.noname-c con 0x7fb3a013f710 addr v1:172.21.15.155:6790/0 2024-03-25T18:31:12.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.574+0000 7fb3a870e640 10 monclient(hunting): picked mon.noname-a con 0x7fb3a01430c0 addr v1:172.21.15.155:6789/0 2024-03-25T18:31:12.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.574+0000 7fb3a870e640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3a0138100 con 0x7fb3a01430c0 2024-03-25T18:31:12.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.574+0000 7fb3a870e640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3a0139890 con 0x7fb3a00a98f0 2024-03-25T18:31:12.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.574+0000 7fb3a870e640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3a0138a10 con 0x7fb3a013f710 2024-03-25T18:31:12.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.574+0000 7fb3a870e640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:12.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.574+0000 7fb3a870e640 10 monclient(hunting): authenticate will time out at 7918.884377s 2024-03-25T18:31:12.586 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.585+0000 7fb3a5ca9640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fb3a013f710 0x7fb3a01d7620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:33742/0 (socket says 172.21.15.155:33742) 2024-03-25T18:31:12.586 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.585+0000 7fb3a5ca9640 1 -- 172.21.15.155:0/508477992 learned_addr learned my addr 172.21.15.155:0/508477992 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:31:12.587 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.586+0000 7fb396ffd640 10 client.?.objecter ms_handle_connect 0x7fb3a013f710 2024-03-25T18:31:12.587 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.586+0000 7fb396ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:12.591 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.590+0000 7fb396ffd640 10 client.?.objecter ms_handle_connect 0x7fb3a00a98f0 2024-03-25T18:31:12.591 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.590+0000 7fb396ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:12.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.592+0000 7fb396ffd640 10 client.?.objecter ms_handle_connect 0x7fb3a01430c0 2024-03-25T18:31:12.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.592+0000 7fb396ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:12.596 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.595+0000 7fb396ffd640 1 -- 172.21.15.155:0/508477992 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1510023800 0 0) 0x7fb3a0138a10 con 0x7fb3a013f710 2024-03-25T18:31:12.596 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.595+0000 7fb396ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:31:12.596 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.595+0000 7fb396ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:12.596 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.595+0000 7fb396ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:12.597 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.595+0000 7fb396ffd640 10 monclient(hunting): my global_id is 4523 2024-03-25T18:31:12.597 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.595+0000 7fb396ffd640 1 -- 172.21.15.155:0/508477992 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb384004eb0 con 0x7fb3a013f710 2024-03-25T18:31:12.604 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.603+0000 7fb396ffd640 1 -- 172.21.15.155:0/508477992 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2574523738 0 0) 0x7fb3a0139890 con 0x7fb3a00a98f0 2024-03-25T18:31:12.604 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.603+0000 7fb396ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:31:12.604 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.603+0000 7fb396ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:12.604 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.603+0000 7fb396ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:12.604 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.603+0000 7fb396ffd640 10 monclient(hunting): my global_id is 4510 2024-03-25T18:31:12.604 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.603+0000 7fb396ffd640 1 -- 172.21.15.155:0/508477992 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3a0138a10 con 0x7fb3a00a98f0 2024-03-25T18:31:12.608 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.607+0000 7fb396ffd640 1 -- 172.21.15.155:0/508477992 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3529476393 0 0) 0x7fb384004eb0 con 0x7fb3a013f710 2024-03-25T18:31:12.608 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.607+0000 7fb396ffd640 1 -- 172.21.15.155:0/508477992 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb3a0139890 con 0x7fb3a013f710 2024-03-25T18:31:12.608 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.607+0000 7fb396ffd640 1 -- 172.21.15.155:0/508477992 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fb3900019b0 con 0x7fb3a013f710 2024-03-25T18:31:12.608 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.607+0000 7fb396ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:12.608 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.607+0000 7fb396ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:31:12.608 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.607+0000 7fb396ffd640 10 monclient(hunting): dump: 2024-03-25T18:31:12.608 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:31:12.609 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:31:12.609 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:12.609 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:12.609 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:12.609 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:31:12.609 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:31:12.609 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:31:12.609 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:31:12.609 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:31:12.609 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.607+0000 7fb396ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:31:12.614 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.612+0000 7fb396ffd640 1 -- 172.21.15.155:0/508477992 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3435602251 0 0) 0x7fb3a0138100 con 0x7fb3a01430c0 2024-03-25T18:31:12.614 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.612+0000 7fb396ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:31:12.614 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.612+0000 7fb396ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:12.614 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.612+0000 7fb396ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:12.614 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.612+0000 7fb396ffd640 10 monclient(hunting): my global_id is 4515 2024-03-25T18:31:12.614 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.612+0000 7fb396ffd640 1 -- 172.21.15.155:0/508477992 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb384004eb0 con 0x7fb3a01430c0 2024-03-25T18:31:12.626 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.625+0000 7fb396ffd640 1 -- 172.21.15.155:0/508477992 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3097644465 0 0) 0x7fb3a0139890 con 0x7fb3a013f710 2024-03-25T18:31:12.627 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.625+0000 7fb396ffd640 1 -- 172.21.15.155:0/508477992 >> v1:172.21.15.167:6789/0 conn(0x7fb3a00a98f0 legacy=0x7fb3a01379f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:12.627 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.625+0000 7fb396ffd640 1 -- 172.21.15.155:0/508477992 >> v1:172.21.15.155:6789/0 conn(0x7fb3a01430c0 legacy=0x7fb3a01d7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:12.627 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.625+0000 7fb396ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:31:12.627 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.625+0000 7fb396ffd640 1 monclient: found mon.c 2024-03-25T18:31:12.627 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.625+0000 7fb396ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:12.627 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.625+0000 7fb396ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:31:12.627 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.625+0000 7fb396ffd640 1 -- 172.21.15.155:0/508477992 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3a01382b0 con 0x7fb3a013f710 2024-03-25T18:31:12.627 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.625+0000 7fb396ffd640 10 monclient: _finish_auth 0 2024-03-25T18:31:12.627 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.625+0000 7fb396ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:31:12.627 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.625+0000 7fb396ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:12.627 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.625+0000 7fb3a870e640 5 monclient: authenticate success, global_id 4523 2024-03-25T18:31:12.627 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.625+0000 7fb3a870e640 10 monclient: _renew_subs 2024-03-25T18:31:12.627 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.625+0000 7fb3a870e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:31:12.627 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.625+0000 7fb3a870e640 1 -- 172.21.15.155:0/508477992 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3a0138560 con 0x7fb3a013f710 2024-03-25T18:31:12.627 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.625+0000 7fb3a870e640 10 client.4523.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:31:12.628 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.625+0000 7fb3a870e640 10 monclient: _renew_subs 2024-03-25T18:31:12.628 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.625+0000 7fb3a870e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:31:12.628 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.625+0000 7fb3a870e640 1 -- 172.21.15.155:0/508477992 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3a01e31c0 con 0x7fb3a013f710 2024-03-25T18:31:12.628 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.625+0000 7fb3a870e640 1 librados: init done 2024-03-25T18:31:12.628 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.626+0000 7fb3a870e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:12.628 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.626+0000 7fb3a870e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:12.629 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.626+0000 7fb3a870e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:31:12.629 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.626+0000 7fb3a870e640 1 -- 172.21.15.155:0/508477992 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3a0138ba0 con 0x7fb3a013f710 2024-03-25T18:31:12.919 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.917+0000 7fb396ffd640 1 -- 172.21.15.155:0/508477992 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb390003610 con 0x7fb3a013f710 2024-03-25T18:31:12.919 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.917+0000 7fb396ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:12.919 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.917+0000 7fb396ffd640 1 -- 172.21.15.155:0/508477992 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fb3900039a0 con 0x7fb3a013f710 2024-03-25T18:31:12.919 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.917+0000 7fb396ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:12.919 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.917+0000 7fb396ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:31:12.919 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.917+0000 7fb396ffd640 10 monclient: dump: 2024-03-25T18:31:12.919 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:31:12.919 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:31:12.919 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:12.919 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:12.919 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:12.920 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:31:12.920 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:31:12.920 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:31:12.920 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:31:12.920 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:31:12.920 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.917+0000 7fb3a4ca7640 4 set_mon_vals no callback set 2024-03-25T18:31:12.920 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.918+0000 7fb396ffd640 1 -- 172.21.15.155:0/508477992 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7fb390003bc0 con 0x7fb3a013f710 2024-03-25T18:31:12.920 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.919+0000 7fb3a64aa640 10 monclient: get_auth_request con 0x7fb38406de00 auth_method 0 2024-03-25T18:31:12.921 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.919+0000 7fb396ffd640 10 client.4523.objecter ms_handle_connect 0x7fb38406de00 2024-03-25T18:31:12.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.926+0000 7fb396ffd640 1 -- 172.21.15.155:0/508477992 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7fb390054780 con 0x7fb3a013f710 2024-03-25T18:31:12.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.926+0000 7fb396ffd640 10 client.4523.objecter ms_dispatch 0x7fb3a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:31:12.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.926+0000 7fb396ffd640 3 client.4523.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:31:12.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.926+0000 7fb396ffd640 3 client.4523.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:31:12.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.926+0000 7fb396ffd640 20 client.4523.objecter dump_active .. 0 homeless 2024-03-25T18:31:12.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.927+0000 7fb396ffd640 1 -- 172.21.15.155:0/508477992 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb390050290 con 0x7fb3a013f710 2024-03-25T18:31:12.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.927+0000 7fb396ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:12.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:12.927+0000 7fb396ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:31:13.176 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:13.175+0000 7fb3a870e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:13.176 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:13.175+0000 7fb3a870e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:13.176 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:13.175+0000 7fb3a870e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:31:13.176 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:13.175+0000 7fb3a870e640 1 -- 172.21.15.155:0/508477992 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb3a00627b0 con 0x7fb3a013f710 2024-03-25T18:31:13.324 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:13.323+0000 7fb396ffd640 1 -- 172.21.15.155:0/508477992 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7fb3900521c0 con 0x7fb3a013f710 2024-03-25T18:31:13.324 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:13.323+0000 7fb396ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:13.324 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:13.323+0000 7fb396ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:31:13.325 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:31:13.325 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:31:13.328 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:13.326+0000 7fb3a870e640 10 librados: watch_flush enter 2024-03-25T18:31:13.328 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:13.326+0000 7fb3a870e640 10 librados: watch_flush exit 2024-03-25T18:31:13.328 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:13.326+0000 7fb3a870e640 20 client.4523.objecter shutdown clearing up homeless session... 2024-03-25T18:31:13.328 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:13.326+0000 7fb3a870e640 10 client.4523.objecter successfully canceled tick 2024-03-25T18:31:13.328 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:13.326+0000 7fb3a870e640 1 -- 172.21.15.155:0/508477992 >> v1:172.21.15.167:6800/2919070503 conn(0x7fb38406de00 legacy=0x7fb3840702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:13.328 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:13.326+0000 7fb3a870e640 10 monclient: shutdown 2024-03-25T18:31:13.328 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:13.326+0000 7fb3a870e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:13.328 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:13.326+0000 7fb3a870e640 1 -- 172.21.15.155:0/508477992 >> v1:172.21.15.155:6790/0 conn(0x7fb3a013f710 legacy=0x7fb3a01d7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:13.328 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:13.326+0000 7fb3a870e640 1 -- 172.21.15.155:0/508477992 shutdown_connections 2024-03-25T18:31:13.328 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:13.326+0000 7fb3a870e640 1 -- 172.21.15.155:0/508477992 >> 172.21.15.155:0/508477992 conn(0x7fb3a00b7ea0 msgr2=0x7fb3a0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:13.329 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:13.326+0000 7fb3a870e640 1 -- 172.21.15.155:0/508477992 shutdown_connections 2024-03-25T18:31:13.329 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:13.326+0000 7fb3a870e640 1 -- 172.21.15.155:0/508477992 wait complete. 2024-03-25T18:31:13.329 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:13.326+0000 7fb3a870e640 1 librados: shutdown 2024-03-25T18:31:13.340 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:31:20.148 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:31:20.268 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.266+0000 7f0329ee6640 10 monclient: get_monmap_and_config 2024-03-25T18:31:20.268 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.266+0000 7f0329ee6640 10 monclient: build_initial_monmap 2024-03-25T18:31:20.268 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.266+0000 7f0329ee6640 10 monclient: monmap: 2024-03-25T18:31:20.268 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:31:20.268 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:31:20.268 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:31:20.267445+0000 2024-03-25T18:31:20.268 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:31:20.267445+0000 2024-03-25T18:31:20.268 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:20.268 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:31:20.268 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:31:20.269 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:31:20.269 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:31:20.269 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:31:20.270 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.267+0000 7f0329ee6640 1 Processor -- start 2024-03-25T18:31:20.270 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.267+0000 7f0329ee6640 1 -- start start 2024-03-25T18:31:20.270 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.267+0000 7f0329ee6640 10 monclient: init 2024-03-25T18:31:20.270 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.268+0000 7f0329ee6640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:20.270 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.268+0000 7f0329ee6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:31:20.270 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.268+0000 7f0329ee6640 10 monclient(hunting): picked mon.noname-a con 0x7f03240a98f0 addr v1:172.21.15.155:6789/0 2024-03-25T18:31:20.271 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.268+0000 7f0329ee6640 10 monclient(hunting): picked mon.noname-b con 0x7f032413f710 addr v1:172.21.15.155:6790/0 2024-03-25T18:31:20.271 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.268+0000 7f0329ee6640 10 monclient(hunting): picked mon.noname-c con 0x7f03241430c0 addr v1:172.21.15.167:6789/0 2024-03-25T18:31:20.271 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.268+0000 7f0329ee6640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0324142520 con 0x7f03240a98f0 2024-03-25T18:31:20.271 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.268+0000 7f0329ee6640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03240b28a0 con 0x7f03241430c0 2024-03-25T18:31:20.271 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.268+0000 7f0329ee6640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03240b2a30 con 0x7f032413f710 2024-03-25T18:31:20.271 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.268+0000 7f0329ee6640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:20.271 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.268+0000 7f0329ee6640 10 monclient(hunting): authenticate will time out at 7926.577939s 2024-03-25T18:31:20.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.274+0000 7f03237fe640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f03240a98f0 0x7f03240a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:45480/0 (socket says 172.21.15.155:45480) 2024-03-25T18:31:20.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.274+0000 7f03237fe640 1 -- 172.21.15.155:0/2536109502 learned_addr learned my addr 172.21.15.155:0/2536109502 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:31:20.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.282+0000 7f0322ffd640 1 -- 172.21.15.155:0/2536109502 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2085600934 0 0) 0x7f0324142520 con 0x7f03240a98f0 2024-03-25T18:31:20.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.282+0000 7f0322ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:31:20.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.282+0000 7f0322ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:20.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.282+0000 7f0322ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:20.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.282+0000 7f0322ffd640 10 monclient(hunting): my global_id is 4518 2024-03-25T18:31:20.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.282+0000 7f0322ffd640 1 -- 172.21.15.155:0/2536109502 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0310003520 con 0x7f03240a98f0 2024-03-25T18:31:20.306 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.305+0000 7f0322ffd640 1 -- 172.21.15.155:0/2536109502 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4165216790 0 0) 0x7f0310003520 con 0x7f03240a98f0 2024-03-25T18:31:20.306 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.305+0000 7f0322ffd640 1 -- 172.21.15.155:0/2536109502 >> v1:172.21.15.155:6790/0 conn(0x7f032413f710 legacy=0x7f03241429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:20.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.305+0000 7f0322ffd640 1 -- 172.21.15.155:0/2536109502 >> v1:172.21.15.167:6789/0 conn(0x7f03241430c0 legacy=0x7f032414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:20.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.305+0000 7f0322ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:31:20.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.305+0000 7f0322ffd640 1 monclient: found mon.noname-a 2024-03-25T18:31:20.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.305+0000 7f0322ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:20.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.305+0000 7f0322ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-25T18:31:20.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.305+0000 7f0322ffd640 1 -- 172.21.15.155:0/2536109502 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03240b44c0 con 0x7f03240a98f0 2024-03-25T18:31:20.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.305+0000 7f0322ffd640 10 monclient: _finish_auth 0 2024-03-25T18:31:20.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.305+0000 7f0322ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:31:20.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.305+0000 7f0322ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:20.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.305+0000 7f0322ffd640 1 -- 172.21.15.155:0/2536109502 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f0314001690 con 0x7f03240a98f0 2024-03-25T18:31:20.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.305+0000 7f0322ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:20.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.305+0000 7f0322ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:31:20.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.305+0000 7f0322ffd640 10 monclient: dump: 2024-03-25T18:31:20.308 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:31:20.308 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:31:20.308 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:20.308 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:20.308 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:20.308 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:31:20.308 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:31:20.308 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:31:20.308 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:31:20.308 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:31:20.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.305+0000 7f0329ee6640 5 monclient: authenticate success, global_id 4518 2024-03-25T18:31:20.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.305+0000 7f0329ee6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:20.312 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.310+0000 7f0322ffd640 1 -- 172.21.15.155:0/2536109502 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f03140017f0 con 0x7f03240a98f0 2024-03-25T18:31:20.312 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.310+0000 7f0322ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:20.312 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.310+0000 7f0322ffd640 1 -- 172.21.15.155:0/2536109502 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f0314001c00 con 0x7f03240a98f0 2024-03-25T18:31:20.312 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.310+0000 7f0322ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:20.312 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.310+0000 7f0322ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:31:20.312 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.310+0000 7f0322ffd640 10 monclient: dump: 2024-03-25T18:31:20.312 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:31:20.312 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:31:20.313 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:20.313 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:20.313 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:20.313 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:31:20.313 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:31:20.313 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:31:20.313 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:31:20.313 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:31:20.313 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.310+0000 7f0329ee6640 10 monclient: get_monmap_and_config success 2024-03-25T18:31:20.313 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.310+0000 7f0329ee6640 4 set_mon_vals no callback set 2024-03-25T18:31:20.316 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.311+0000 7f0329ee6640 10 monclient: shutdown 2024-03-25T18:31:20.316 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.311+0000 7f0329ee6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:20.316 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.311+0000 7f0329ee6640 1 -- 172.21.15.155:0/2536109502 >> v1:172.21.15.155:6789/0 conn(0x7f03240a98f0 legacy=0x7f03240a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:20.316 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.311+0000 7f0329ee6640 1 -- 172.21.15.155:0/2536109502 shutdown_connections 2024-03-25T18:31:20.316 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.311+0000 7f0329ee6640 1 -- 172.21.15.155:0/2536109502 >> 172.21.15.155:0/2536109502 conn(0x7f03240b7ea0 msgr2=0x7f03240b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:20.316 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.311+0000 7f0329ee6640 1 -- 172.21.15.155:0/2536109502 shutdown_connections 2024-03-25T18:31:20.316 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.311+0000 7f0329ee6640 1 -- 172.21.15.155:0/2536109502 wait complete. 2024-03-25T18:31:20.316 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.312+0000 7f0329ee6640 10 monclient: build_initial_monmap 2024-03-25T18:31:20.316 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.312+0000 7f0329ee6640 10 monclient: monmap: 2024-03-25T18:31:20.316 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:31:20.316 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:31:20.316 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:31:20.317 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:31:20.317 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:20.317 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:31:20.317 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:31:20.317 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:31:20.317 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:31:20.317 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:31:20.317 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.312+0000 7f0329ee6640 1 librados: starting msgr at 2024-03-25T18:31:20.317 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.312+0000 7f0329ee6640 1 librados: starting objecter 2024-03-25T18:31:20.317 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.312+0000 7f0329ee6640 1 Processor -- start 2024-03-25T18:31:20.317 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.312+0000 7f0329ee6640 1 -- start start 2024-03-25T18:31:20.317 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.312+0000 7f0329ee6640 1 librados: setting wanted keys 2024-03-25T18:31:20.317 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.312+0000 7f0329ee6640 1 librados: calling monclient init 2024-03-25T18:31:20.317 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.312+0000 7f0329ee6640 10 monclient: init 2024-03-25T18:31:20.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.313+0000 7f0329ee6640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:20.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.313+0000 7f0329ee6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:31:20.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.313+0000 7f0329ee6640 10 monclient(hunting): picked mon.noname-a con 0x7f03240a98f0 addr v1:172.21.15.155:6789/0 2024-03-25T18:31:20.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.313+0000 7f0329ee6640 10 monclient(hunting): picked mon.noname-c con 0x7f032413f710 addr v1:172.21.15.155:6790/0 2024-03-25T18:31:20.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.313+0000 7f0329ee6640 10 monclient(hunting): picked mon.noname-b con 0x7f03241430c0 addr v1:172.21.15.167:6789/0 2024-03-25T18:31:20.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.313+0000 7f0329ee6640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0324135ac0 con 0x7f03240a98f0 2024-03-25T18:31:20.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.313+0000 7f0329ee6640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0324137030 con 0x7f03241430c0 2024-03-25T18:31:20.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.313+0000 7f0329ee6640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03241372a0 con 0x7f032413f710 2024-03-25T18:31:20.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.313+0000 7f0329ee6640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:20.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.313+0000 7f0329ee6640 10 monclient(hunting): authenticate will time out at 7926.623329s 2024-03-25T18:31:20.326 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.325+0000 7f031bfff640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f032413f710 0x7f03241379a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:56040/0 (socket says 172.21.15.155:56040) 2024-03-25T18:31:20.326 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.325+0000 7f031bfff640 1 -- 172.21.15.155:0/914659652 learned_addr learned my addr 172.21.15.155:0/914659652 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:31:20.328 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.327+0000 7f0320ff9640 10 client.?.objecter ms_handle_connect 0x7f032413f710 2024-03-25T18:31:20.328 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.327+0000 7f0320ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:20.331 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.330+0000 7f0320ff9640 10 client.?.objecter ms_handle_connect 0x7f03241430c0 2024-03-25T18:31:20.331 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.330+0000 7f0320ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:20.333 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.332+0000 7f0320ff9640 10 client.?.objecter ms_handle_connect 0x7f03240a98f0 2024-03-25T18:31:20.334 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.332+0000 7f0320ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:20.336 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.335+0000 7f0320ff9640 1 -- 172.21.15.155:0/914659652 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 987364190 0 0) 0x7f03241372a0 con 0x7f032413f710 2024-03-25T18:31:20.337 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.335+0000 7f0320ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:31:20.337 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.335+0000 7f0320ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:20.337 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.335+0000 7f0320ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:20.337 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.335+0000 7f0320ff9640 10 monclient(hunting): my global_id is 4529 2024-03-25T18:31:20.337 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.335+0000 7f0320ff9640 1 -- 172.21.15.155:0/914659652 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0304004eb0 con 0x7f032413f710 2024-03-25T18:31:20.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.341+0000 7f0320ff9640 1 -- 172.21.15.155:0/914659652 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2111887714 0 0) 0x7f0324135ac0 con 0x7f03240a98f0 2024-03-25T18:31:20.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.341+0000 7f0320ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:31:20.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.341+0000 7f0320ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:20.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.341+0000 7f0320ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:20.343 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.341+0000 7f0320ff9640 10 monclient(hunting): my global_id is 4521 2024-03-25T18:31:20.343 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.341+0000 7f0320ff9640 1 -- 172.21.15.155:0/914659652 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03241372a0 con 0x7f03240a98f0 2024-03-25T18:31:20.350 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.348+0000 7f0320ff9640 1 -- 172.21.15.155:0/914659652 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3659979701 0 0) 0x7f0324137030 con 0x7f03241430c0 2024-03-25T18:31:20.350 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.348+0000 7f0320ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:31:20.350 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.348+0000 7f0320ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:20.350 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.348+0000 7f0320ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:20.350 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.348+0000 7f0320ff9640 10 monclient(hunting): my global_id is 4516 2024-03-25T18:31:20.350 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.348+0000 7f0320ff9640 1 -- 172.21.15.155:0/914659652 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0324135ac0 con 0x7f03241430c0 2024-03-25T18:31:20.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.356+0000 7f0320ff9640 1 -- 172.21.15.155:0/914659652 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 314878239 0 0) 0x7f0324135ac0 con 0x7f03241430c0 2024-03-25T18:31:20.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.356+0000 7f0320ff9640 1 -- 172.21.15.155:0/914659652 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0324137030 con 0x7f03241430c0 2024-03-25T18:31:20.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.356+0000 7f0320ff9640 1 -- 172.21.15.155:0/914659652 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f031c001a90 con 0x7f03241430c0 2024-03-25T18:31:20.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.356+0000 7f0320ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:20.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.356+0000 7f0320ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:31:20.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.356+0000 7f0320ff9640 10 monclient(hunting): dump: 2024-03-25T18:31:20.358 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:31:20.358 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:31:20.358 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:20.359 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:20.359 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:20.359 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:31:20.359 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:31:20.359 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:31:20.359 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:31:20.359 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:31:20.359 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.356+0000 7f0320ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:31:20.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.357+0000 7f0320ff9640 1 -- 172.21.15.155:0/914659652 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2024051504 0 0) 0x7f03241372a0 con 0x7f03240a98f0 2024-03-25T18:31:20.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.357+0000 7f0320ff9640 1 -- 172.21.15.155:0/914659652 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0324135ac0 con 0x7f03240a98f0 2024-03-25T18:31:20.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.357+0000 7f0320ff9640 1 -- 172.21.15.155:0/914659652 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f0314001b00 con 0x7f03240a98f0 2024-03-25T18:31:20.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.357+0000 7f0320ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:20.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.357+0000 7f0320ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:31:20.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.357+0000 7f0320ff9640 10 monclient(hunting): dump: 2024-03-25T18:31:20.360 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:31:20.360 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:31:20.360 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:20.360 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:20.361 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:20.361 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:31:20.361 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:31:20.361 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:31:20.361 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:31:20.361 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:31:20.363 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.361+0000 7f0320ff9640 1 -- 172.21.15.155:0/914659652 <== mon.1 v1:172.21.15.167:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1252185579 0 0) 0x7f0324137030 con 0x7f03241430c0 2024-03-25T18:31:20.363 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.361+0000 7f0320ff9640 1 -- 172.21.15.155:0/914659652 >> v1:172.21.15.155:6790/0 conn(0x7f032413f710 legacy=0x7f03241379a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:20.363 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.361+0000 7f0320ff9640 1 -- 172.21.15.155:0/914659652 >> v1:172.21.15.155:6789/0 conn(0x7f03240a98f0 legacy=0x7f0324133440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:20.363 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.361+0000 7f0320ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:31:20.363 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.361+0000 7f0320ff9640 1 monclient: found mon.b 2024-03-25T18:31:20.364 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.361+0000 7f0320ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:20.364 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.361+0000 7f0320ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:31:20.364 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.361+0000 7f0320ff9640 1 -- 172.21.15.155:0/914659652 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0324137580 con 0x7f03241430c0 2024-03-25T18:31:20.364 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.361+0000 7f0320ff9640 10 monclient: _finish_auth 0 2024-03-25T18:31:20.364 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.361+0000 7f0320ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:31:20.364 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.361+0000 7f0320ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:20.364 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.361+0000 7f0329ee6640 5 monclient: authenticate success, global_id 4516 2024-03-25T18:31:20.364 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.361+0000 7f0329ee6640 10 monclient: _renew_subs 2024-03-25T18:31:20.364 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.361+0000 7f0329ee6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:31:20.364 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.361+0000 7f0329ee6640 1 -- 172.21.15.155:0/914659652 --> v1:172.21.15.167:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03241363b0 con 0x7f03241430c0 2024-03-25T18:31:20.364 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.361+0000 7f0329ee6640 10 client.4516.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:31:20.364 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.361+0000 7f0329ee6640 10 monclient: _renew_subs 2024-03-25T18:31:20.365 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.361+0000 7f0329ee6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:31:20.365 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.361+0000 7f0329ee6640 1 -- 172.21.15.155:0/914659652 --> v1:172.21.15.167:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f03241353a0 con 0x7f03241430c0 2024-03-25T18:31:20.365 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.361+0000 7f0329ee6640 1 librados: init done 2024-03-25T18:31:20.365 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.362+0000 7f0329ee6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:20.365 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.362+0000 7f0329ee6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:20.366 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.362+0000 7f0329ee6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:31:20.366 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.362+0000 7f0329ee6640 1 -- 172.21.15.155:0/914659652 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0324130d90 con 0x7f03241430c0 2024-03-25T18:31:20.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.636+0000 7f0320ff9640 1 -- 172.21.15.155:0/914659652 <== mon.1 v1:172.21.15.167:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f031c001fa0 con 0x7f03241430c0 2024-03-25T18:31:20.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.636+0000 7f0320ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:20.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.636+0000 7f0320ff9640 1 -- 172.21.15.155:0/914659652 <== mon.1 v1:172.21.15.167:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f031c003970 con 0x7f03241430c0 2024-03-25T18:31:20.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.636+0000 7f0320ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:20.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.636+0000 7f0320ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:31:20.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.636+0000 7f0320ff9640 10 monclient: dump: 2024-03-25T18:31:20.638 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:31:20.638 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:31:20.638 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:20.638 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:20.638 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:20.638 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:31:20.638 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:31:20.639 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:31:20.639 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:31:20.639 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:31:20.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.636+0000 7f03227fc640 4 set_mon_vals no callback set 2024-03-25T18:31:20.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.637+0000 7f0320ff9640 1 -- 172.21.15.155:0/914659652 <== mon.1 v1:172.21.15.167:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7f031c003b90 con 0x7f03241430c0 2024-03-25T18:31:20.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.638+0000 7f03237fe640 10 monclient: get_auth_request con 0x7f030406df80 auth_method 0 2024-03-25T18:31:20.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.638+0000 7f0320ff9640 10 client.4516.objecter ms_handle_connect 0x7f030406df80 2024-03-25T18:31:20.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.644+0000 7f0320ff9640 1 -- 172.21.15.155:0/914659652 <== mon.1 v1:172.21.15.167:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7f031c053620 con 0x7f03241430c0 2024-03-25T18:31:20.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.644+0000 7f0320ff9640 10 client.4516.objecter ms_dispatch 0x7f0324000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:31:20.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.644+0000 7f0320ff9640 3 client.4516.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:31:20.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.644+0000 7f0320ff9640 3 client.4516.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:31:20.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.644+0000 7f0320ff9640 20 client.4516.objecter dump_active .. 0 homeless 2024-03-25T18:31:20.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.644+0000 7f0320ff9640 1 -- 172.21.15.155:0/914659652 <== mon.1 v1:172.21.15.167:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f031c04f130 con 0x7f03241430c0 2024-03-25T18:31:20.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.644+0000 7f0320ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:20.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.644+0000 7f0320ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:31:20.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.893+0000 7f0329ee6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:20.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.893+0000 7f0329ee6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:20.895 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.893+0000 7f0329ee6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:31:20.895 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.893+0000 7f0329ee6640 1 -- 172.21.15.155:0/914659652 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0324074cc0 con 0x7f03241430c0 2024-03-25T18:31:20.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.927+0000 7f0320ff9640 1 -- 172.21.15.155:0/914659652 <== mon.1 v1:172.21.15.167:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7f031c051060 con 0x7f03241430c0 2024-03-25T18:31:20.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.927+0000 7f0320ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:20.929 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.927+0000 7f0320ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:31:20.929 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:31:20.929 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:31:20.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.929+0000 7f0329ee6640 10 librados: watch_flush enter 2024-03-25T18:31:20.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.929+0000 7f0329ee6640 10 librados: watch_flush exit 2024-03-25T18:31:20.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.929+0000 7f0329ee6640 20 client.4516.objecter shutdown clearing up homeless session... 2024-03-25T18:31:20.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.929+0000 7f0329ee6640 10 client.4516.objecter successfully canceled tick 2024-03-25T18:31:20.932 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.930+0000 7f0329ee6640 1 -- 172.21.15.155:0/914659652 >> v1:172.21.15.167:6800/2919070503 conn(0x7f030406df80 legacy=0x7f0304070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:20.932 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.930+0000 7f0329ee6640 10 monclient: shutdown 2024-03-25T18:31:20.932 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.930+0000 7f0329ee6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:20.932 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.930+0000 7f0329ee6640 1 -- 172.21.15.155:0/914659652 >> v1:172.21.15.167:6789/0 conn(0x7f03241430c0 legacy=0x7f03241e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:20.932 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.930+0000 7f0329ee6640 1 -- 172.21.15.155:0/914659652 shutdown_connections 2024-03-25T18:31:20.932 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.930+0000 7f0329ee6640 1 -- 172.21.15.155:0/914659652 >> 172.21.15.155:0/914659652 conn(0x7f03240b7ea0 msgr2=0x7f0324074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:20.932 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.930+0000 7f0329ee6640 1 -- 172.21.15.155:0/914659652 shutdown_connections 2024-03-25T18:31:20.932 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.930+0000 7f0329ee6640 1 -- 172.21.15.155:0/914659652 wait complete. 2024-03-25T18:31:20.932 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:20.930+0000 7f0329ee6640 1 librados: shutdown 2024-03-25T18:31:20.943 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:31:26.663 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:07:00.267 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:26.690 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:07:00.294 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:26.719 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:07:00.297 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:26.720 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:07:00.324 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:26.732 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:07:00.311 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:26.741 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:07:00.346 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:26.746 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:07:00.351 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:26.759 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:07:00.338 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:26.782 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:07:00.387 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:27.749 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:31:27.862 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.860+0000 7f5110df4640 10 monclient: get_monmap_and_config 2024-03-25T18:31:27.862 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.860+0000 7f5110df4640 10 monclient: build_initial_monmap 2024-03-25T18:31:27.862 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.860+0000 7f5110df4640 10 monclient: monmap: 2024-03-25T18:31:27.862 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:31:27.862 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:31:27.862 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:31:27.861322+0000 2024-03-25T18:31:27.862 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:31:27.861322+0000 2024-03-25T18:31:27.862 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:27.863 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:31:27.863 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:31:27.863 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:31:27.863 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:31:27.863 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:31:27.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.861+0000 7f5110df4640 1 Processor -- start 2024-03-25T18:31:27.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.861+0000 7f5110df4640 1 -- start start 2024-03-25T18:31:27.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.861+0000 7f5110df4640 10 monclient: init 2024-03-25T18:31:27.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.861+0000 7f5110df4640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:27.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.861+0000 7f5110df4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:31:27.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.862+0000 7f5110df4640 10 monclient(hunting): picked mon.noname-a con 0x7f510c0a98f0 addr v1:172.21.15.155:6789/0 2024-03-25T18:31:27.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.862+0000 7f5110df4640 10 monclient(hunting): picked mon.noname-b con 0x7f510c13f710 addr v1:172.21.15.155:6790/0 2024-03-25T18:31:27.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.862+0000 7f5110df4640 10 monclient(hunting): picked mon.noname-c con 0x7f510c1430c0 addr v1:172.21.15.167:6789/0 2024-03-25T18:31:27.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.862+0000 7f5110df4640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f510c142520 con 0x7f510c0a98f0 2024-03-25T18:31:27.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.862+0000 7f5110df4640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f510c0b28a0 con 0x7f510c1430c0 2024-03-25T18:31:27.865 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.862+0000 7f5110df4640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f510c0b2a30 con 0x7f510c13f710 2024-03-25T18:31:27.865 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.862+0000 7f5110df4640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:27.865 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.862+0000 7f5110df4640 10 monclient(hunting): authenticate will time out at 7934.171813s 2024-03-25T18:31:27.881 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.880+0000 7f510ad9d640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f510c1430c0 0x7f510c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:34830/0 (socket says 172.21.15.155:34830) 2024-03-25T18:31:27.882 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.880+0000 7f510ad9d640 1 -- 172.21.15.155:0/558329772 learned_addr learned my addr 172.21.15.155:0/558329772 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:31:27.890 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.889+0000 7f510959a640 1 -- 172.21.15.155:0/558329772 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2677386434 0 0) 0x7f510c0b28a0 con 0x7f510c1430c0 2024-03-25T18:31:27.890 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.889+0000 7f510959a640 10 monclient(hunting): _negotiate 2024-03-25T18:31:27.891 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.889+0000 7f510959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:27.891 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.889+0000 7f510959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:27.891 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.889+0000 7f510959a640 10 monclient(hunting): my global_id is 4519 2024-03-25T18:31:27.891 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.889+0000 7f510959a640 1 -- 172.21.15.155:0/558329772 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50f4003520 con 0x7f510c1430c0 2024-03-25T18:31:27.907 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.905+0000 7f510959a640 1 -- 172.21.15.155:0/558329772 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2738659793 0 0) 0x7f510c142520 con 0x7f510c0a98f0 2024-03-25T18:31:27.907 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.905+0000 7f510959a640 10 monclient(hunting): _negotiate 2024-03-25T18:31:27.907 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.905+0000 7f510959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:27.907 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.905+0000 7f510959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:27.907 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.905+0000 7f510959a640 10 monclient(hunting): my global_id is 4524 2024-03-25T18:31:27.907 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.905+0000 7f510959a640 1 -- 172.21.15.155:0/558329772 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f510c0b28a0 con 0x7f510c0a98f0 2024-03-25T18:31:27.907 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.905+0000 7f510959a640 1 -- 172.21.15.155:0/558329772 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3848697340 0 0) 0x7f50f4003520 con 0x7f510c1430c0 2024-03-25T18:31:27.907 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.905+0000 7f510959a640 1 -- 172.21.15.155:0/558329772 >> v1:172.21.15.155:6790/0 conn(0x7f510c13f710 legacy=0x7f510c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:27.908 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.905+0000 7f510959a640 1 -- 172.21.15.155:0/558329772 >> v1:172.21.15.155:6789/0 conn(0x7f510c0a98f0 legacy=0x7f510c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:27.908 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.905+0000 7f510959a640 10 monclient: _finish_hunting 0 2024-03-25T18:31:27.908 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.905+0000 7f510959a640 1 monclient: found mon.noname-c 2024-03-25T18:31:27.908 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.905+0000 7f510959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:27.908 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.905+0000 7f510959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.167:6789/0 2024-03-25T18:31:27.908 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.905+0000 7f510959a640 1 -- 172.21.15.155:0/558329772 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f510c0b44c0 con 0x7f510c1430c0 2024-03-25T18:31:27.908 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.905+0000 7f510959a640 10 monclient: _finish_auth 0 2024-03-25T18:31:27.908 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.905+0000 7f510959a640 10 monclient: _check_auth_tickets 2024-03-25T18:31:27.908 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.905+0000 7f510959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:27.908 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.905+0000 7f5110df4640 5 monclient: authenticate success, global_id 4519 2024-03-25T18:31:27.908 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.905+0000 7f510959a640 1 -- 172.21.15.155:0/558329772 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f5104001a10 con 0x7f510c1430c0 2024-03-25T18:31:27.908 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.905+0000 7f5110df4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:27.908 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.905+0000 7f510959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:27.908 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.905+0000 7f510959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:31:27.909 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.905+0000 7f510959a640 10 monclient: dump: 2024-03-25T18:31:27.909 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:31:27.909 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:31:27.909 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:27.909 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:27.909 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:27.909 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:31:27.909 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:31:27.909 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:31:27.909 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:31:27.909 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:31:27.909 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.905+0000 7f5110df4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:27.919 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.918+0000 7f510959a640 1 -- 172.21.15.155:0/558329772 <== mon.1 v1:172.21.15.167:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5104001b70 con 0x7f510c1430c0 2024-03-25T18:31:27.919 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.918+0000 7f510959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:27.919 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.918+0000 7f510959a640 1 -- 172.21.15.155:0/558329772 <== mon.1 v1:172.21.15.167:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f5104001f80 con 0x7f510c1430c0 2024-03-25T18:31:27.919 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.918+0000 7f510959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:27.919 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.918+0000 7f510959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:31:27.919 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.918+0000 7f510959a640 10 monclient: dump: 2024-03-25T18:31:27.919 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:31:27.919 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:31:27.919 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:27.920 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:27.920 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:27.920 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:31:27.920 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:31:27.920 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:31:27.920 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:31:27.920 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:31:27.920 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.918+0000 7f5110df4640 10 monclient: get_monmap_and_config success 2024-03-25T18:31:27.920 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.918+0000 7f5110df4640 4 set_mon_vals no callback set 2024-03-25T18:31:27.921 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.918+0000 7f5110df4640 10 monclient: shutdown 2024-03-25T18:31:27.922 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.918+0000 7f5110df4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:27.922 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.918+0000 7f5110df4640 1 -- 172.21.15.155:0/558329772 >> v1:172.21.15.167:6789/0 conn(0x7f510c1430c0 legacy=0x7f510c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:27.922 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.918+0000 7f5110df4640 1 -- 172.21.15.155:0/558329772 shutdown_connections 2024-03-25T18:31:27.922 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.919+0000 7f5110df4640 1 -- 172.21.15.155:0/558329772 >> 172.21.15.155:0/558329772 conn(0x7f510c0b7ea0 msgr2=0x7f510c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:27.922 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.919+0000 7f5110df4640 1 -- 172.21.15.155:0/558329772 shutdown_connections 2024-03-25T18:31:27.922 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.919+0000 7f5110df4640 1 -- 172.21.15.155:0/558329772 wait complete. 2024-03-25T18:31:27.922 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.920+0000 7f5110df4640 10 monclient: build_initial_monmap 2024-03-25T18:31:27.922 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.920+0000 7f5110df4640 10 monclient: monmap: 2024-03-25T18:31:27.922 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:31:27.922 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:31:27.922 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:31:27.922 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:31:27.922 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:27.923 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:31:27.923 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:31:27.923 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:31:27.923 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:31:27.923 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:31:27.923 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.920+0000 7f5110df4640 1 librados: starting msgr at 2024-03-25T18:31:27.923 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.920+0000 7f5110df4640 1 librados: starting objecter 2024-03-25T18:31:27.923 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.920+0000 7f5110df4640 1 Processor -- start 2024-03-25T18:31:27.923 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.920+0000 7f5110df4640 1 -- start start 2024-03-25T18:31:27.923 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.920+0000 7f5110df4640 1 librados: setting wanted keys 2024-03-25T18:31:27.923 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.920+0000 7f5110df4640 1 librados: calling monclient init 2024-03-25T18:31:27.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.920+0000 7f5110df4640 10 monclient: init 2024-03-25T18:31:27.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.921+0000 7f5110df4640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:27.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.921+0000 7f5110df4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:31:27.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.921+0000 7f5110df4640 10 monclient(hunting): picked mon.noname-a con 0x7f510c0a98f0 addr v1:172.21.15.155:6789/0 2024-03-25T18:31:27.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.921+0000 7f5110df4640 10 monclient(hunting): picked mon.noname-b con 0x7f510c13f710 addr v1:172.21.15.167:6789/0 2024-03-25T18:31:27.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.921+0000 7f5110df4640 10 monclient(hunting): picked mon.noname-c con 0x7f510c1430c0 addr v1:172.21.15.155:6790/0 2024-03-25T18:31:27.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.921+0000 7f5110df4640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f510c1daca0 con 0x7f510c0a98f0 2024-03-25T18:31:27.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.921+0000 7f5110df4640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f510c138ab0 con 0x7f510c13f710 2024-03-25T18:31:27.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.921+0000 7f5110df4640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f510c0aba90 con 0x7f510c1430c0 2024-03-25T18:31:27.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.921+0000 7f5110df4640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:27.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.921+0000 7f5110df4640 10 monclient(hunting): authenticate will time out at 7934.231594s 2024-03-25T18:31:27.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.926+0000 7f510ad9d640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f510c1430c0 0x7f510c1d47b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:60346/0 (socket says 172.21.15.155:60346) 2024-03-25T18:31:27.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.926+0000 7f510ad9d640 1 -- 172.21.15.155:0/2422086335 learned_addr learned my addr 172.21.15.155:0/2422086335 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:31:27.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.930+0000 7f50faffd640 10 client.?.objecter ms_handle_connect 0x7f510c1430c0 2024-03-25T18:31:27.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.930+0000 7f50faffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:27.933 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.932+0000 7f50faffd640 10 client.?.objecter ms_handle_connect 0x7f510c13f710 2024-03-25T18:31:27.933 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.932+0000 7f50faffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:27.940 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.938+0000 7f50faffd640 1 -- 172.21.15.155:0/2422086335 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 718781452 0 0) 0x7f510c138ab0 con 0x7f510c13f710 2024-03-25T18:31:27.940 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.938+0000 7f50faffd640 10 monclient(hunting): _negotiate 2024-03-25T18:31:27.940 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.938+0000 7f50faffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:27.940 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.938+0000 7f50faffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:27.940 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.938+0000 7f50faffd640 10 monclient(hunting): my global_id is 4522 2024-03-25T18:31:27.940 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.938+0000 7f50faffd640 1 -- 172.21.15.155:0/2422086335 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50ec004eb0 con 0x7f510c13f710 2024-03-25T18:31:27.942 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.941+0000 7f50faffd640 1 -- 172.21.15.155:0/2422086335 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2304757747 0 0) 0x7f510c0aba90 con 0x7f510c1430c0 2024-03-25T18:31:27.943 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.941+0000 7f50faffd640 10 monclient(hunting): _negotiate 2024-03-25T18:31:27.943 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.941+0000 7f50faffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:27.943 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.941+0000 7f50faffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:27.943 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.941+0000 7f50faffd640 10 monclient(hunting): my global_id is 4535 2024-03-25T18:31:27.943 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.941+0000 7f50faffd640 1 -- 172.21.15.155:0/2422086335 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f510c138ab0 con 0x7f510c1430c0 2024-03-25T18:31:27.945 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.944+0000 7f50faffd640 10 client.?.objecter ms_handle_connect 0x7f510c0a98f0 2024-03-25T18:31:27.945 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.944+0000 7f50faffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:27.952 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.950+0000 7f50faffd640 1 -- 172.21.15.155:0/2422086335 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1014274586 0 0) 0x7f50ec004eb0 con 0x7f510c13f710 2024-03-25T18:31:27.952 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.951+0000 7f50faffd640 1 -- 172.21.15.155:0/2422086335 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f510c0aba90 con 0x7f510c13f710 2024-03-25T18:31:27.952 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.951+0000 7f50faffd640 1 -- 172.21.15.155:0/2422086335 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f51000018c0 con 0x7f510c13f710 2024-03-25T18:31:27.952 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.951+0000 7f50faffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:27.952 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.951+0000 7f50faffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:31:27.952 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.951+0000 7f50faffd640 10 monclient(hunting): dump: 2024-03-25T18:31:27.952 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:31:27.953 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:31:27.953 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:27.953 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:27.953 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:27.953 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:31:27.953 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:31:27.953 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:31:27.953 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:31:27.953 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:31:27.953 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.951+0000 7f50faffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:31:27.954 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.951+0000 7f50faffd640 1 -- 172.21.15.155:0/2422086335 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2539389142 0 0) 0x7f510c138ab0 con 0x7f510c1430c0 2024-03-25T18:31:27.954 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.951+0000 7f50faffd640 1 -- 172.21.15.155:0/2422086335 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f50ec004eb0 con 0x7f510c1430c0 2024-03-25T18:31:27.954 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.951+0000 7f50faffd640 1 -- 172.21.15.155:0/2422086335 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f5104001e80 con 0x7f510c1430c0 2024-03-25T18:31:27.954 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.951+0000 7f50faffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:27.954 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.951+0000 7f50faffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:31:27.955 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.951+0000 7f50faffd640 10 monclient(hunting): dump: 2024-03-25T18:31:27.955 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:31:27.955 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:31:27.955 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:27.955 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:27.955 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:27.955 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:31:27.955 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:31:27.955 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:31:27.955 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:31:27.955 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:31:27.960 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.958+0000 7f50faffd640 1 -- 172.21.15.155:0/2422086335 <== mon.1 v1:172.21.15.167:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 9724292 0 0) 0x7f510c0aba90 con 0x7f510c13f710 2024-03-25T18:31:27.960 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.958+0000 7f50faffd640 1 -- 172.21.15.155:0/2422086335 >> v1:172.21.15.155:6790/0 conn(0x7f510c1430c0 legacy=0x7f510c1d47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:27.960 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.958+0000 7f50faffd640 1 -- 172.21.15.155:0/2422086335 >> v1:172.21.15.155:6789/0 conn(0x7f510c0a98f0 legacy=0x7f510c1383a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:27.960 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.958+0000 7f50faffd640 10 monclient: _finish_hunting 0 2024-03-25T18:31:27.960 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.958+0000 7f50faffd640 1 monclient: found mon.b 2024-03-25T18:31:27.960 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.958+0000 7f50faffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:27.960 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.958+0000 7f50faffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:31:27.961 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.958+0000 7f50faffd640 1 -- 172.21.15.155:0/2422086335 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f510c1d41e0 con 0x7f510c13f710 2024-03-25T18:31:27.961 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.958+0000 7f50faffd640 10 monclient: _finish_auth 0 2024-03-25T18:31:27.961 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.958+0000 7f50faffd640 10 monclient: _check_auth_tickets 2024-03-25T18:31:27.961 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.958+0000 7f50faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:27.961 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.958+0000 7f5110df4640 5 monclient: authenticate success, global_id 4522 2024-03-25T18:31:27.961 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.958+0000 7f5110df4640 10 monclient: _renew_subs 2024-03-25T18:31:27.961 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.958+0000 7f5110df4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:31:27.961 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.958+0000 7f5110df4640 1 -- 172.21.15.155:0/2422086335 --> v1:172.21.15.167:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f510c1db840 con 0x7f510c13f710 2024-03-25T18:31:27.961 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.958+0000 7f5110df4640 10 client.4522.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:31:27.961 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.958+0000 7f5110df4640 10 monclient: _renew_subs 2024-03-25T18:31:27.961 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.958+0000 7f5110df4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:31:27.961 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.958+0000 7f5110df4640 1 -- 172.21.15.155:0/2422086335 --> v1:172.21.15.167:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f510c138f00 con 0x7f510c13f710 2024-03-25T18:31:27.961 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.958+0000 7f5110df4640 1 librados: init done 2024-03-25T18:31:27.962 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.960+0000 7f5110df4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:27.962 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.960+0000 7f5110df4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:27.962 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.960+0000 7f5110df4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:31:27.962 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:27.960+0000 7f5110df4640 1 -- 172.21.15.155:0/2422086335 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f510c1dae50 con 0x7f510c13f710 2024-03-25T18:31:28.231 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:28.229+0000 7f50faffd640 1 -- 172.21.15.155:0/2422086335 <== mon.1 v1:172.21.15.167:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f51000035f0 con 0x7f510c13f710 2024-03-25T18:31:28.231 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:28.229+0000 7f50faffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:28.231 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:28.229+0000 7f50faffd640 1 -- 172.21.15.155:0/2422086335 <== mon.1 v1:172.21.15.167:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f5100003760 con 0x7f510c13f710 2024-03-25T18:31:28.231 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:28.229+0000 7f50faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:28.231 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:28.229+0000 7f50faffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:31:28.231 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:28.229+0000 7f50faffd640 10 monclient: dump: 2024-03-25T18:31:28.231 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:31:28.231 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:31:28.231 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:28.231 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:28.232 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:28.232 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:31:28.232 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:31:28.232 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:31:28.232 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:31:28.232 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:31:28.232 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:28.229+0000 7f5108d99640 4 set_mon_vals no callback set 2024-03-25T18:31:28.233 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:28.230+0000 7f50faffd640 1 -- 172.21.15.155:0/2422086335 <== mon.1 v1:172.21.15.167:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7f5100019730 con 0x7f510c13f710 2024-03-25T18:31:28.233 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:28.231+0000 7f510a59c640 10 monclient: get_auth_request con 0x7f50ec06de40 auth_method 0 2024-03-25T18:31:28.233 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:28.231+0000 7f50faffd640 10 client.4522.objecter ms_handle_connect 0x7f50ec06de40 2024-03-25T18:31:28.239 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:28.238+0000 7f50faffd640 1 -- 172.21.15.155:0/2422086335 <== mon.1 v1:172.21.15.167:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7f5100053420 con 0x7f510c13f710 2024-03-25T18:31:28.240 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:28.238+0000 7f50faffd640 10 client.4522.objecter ms_dispatch 0x7f510c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:31:28.240 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:28.238+0000 7f50faffd640 3 client.4522.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:31:28.240 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:28.238+0000 7f50faffd640 3 client.4522.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:31:28.240 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:28.238+0000 7f50faffd640 20 client.4522.objecter dump_active .. 0 homeless 2024-03-25T18:31:28.240 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:28.239+0000 7f50faffd640 1 -- 172.21.15.155:0/2422086335 <== mon.1 v1:172.21.15.167:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f510004ed60 con 0x7f510c13f710 2024-03-25T18:31:28.240 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:28.239+0000 7f50faffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:28.240 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:28.239+0000 7f50faffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:31:28.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:28.492+0000 7f5110df4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:28.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:28.492+0000 7f5110df4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:28.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:28.492+0000 7f5110df4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:31:28.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:28.492+0000 7f5110df4640 1 -- 172.21.15.155:0/2422086335 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f510c0627b0 con 0x7f510c13f710 2024-03-25T18:31:28.515 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:28.514+0000 7f50faffd640 1 -- 172.21.15.155:0/2422086335 <== mon.1 v1:172.21.15.167:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7f5100050e60 con 0x7f510c13f710 2024-03-25T18:31:28.515 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:28.514+0000 7f50faffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:28.515 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:28.514+0000 7f50faffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:31:28.516 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:31:28.516 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:31:28.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:28.516+0000 7f5110df4640 10 librados: watch_flush enter 2024-03-25T18:31:28.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:28.516+0000 7f5110df4640 10 librados: watch_flush exit 2024-03-25T18:31:28.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:28.517+0000 7f5110df4640 20 client.4522.objecter shutdown clearing up homeless session... 2024-03-25T18:31:28.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:28.517+0000 7f5110df4640 10 client.4522.objecter successfully canceled tick 2024-03-25T18:31:28.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:28.517+0000 7f5110df4640 1 -- 172.21.15.155:0/2422086335 >> v1:172.21.15.167:6800/2919070503 conn(0x7f50ec06de40 legacy=0x7f50ec070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:28.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:28.517+0000 7f5110df4640 10 monclient: shutdown 2024-03-25T18:31:28.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:28.517+0000 7f5110df4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:28.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:28.517+0000 7f5110df4640 1 -- 172.21.15.155:0/2422086335 >> v1:172.21.15.167:6789/0 conn(0x7f510c13f710 legacy=0x7f510c1db130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:28.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:28.517+0000 7f5110df4640 1 -- 172.21.15.155:0/2422086335 shutdown_connections 2024-03-25T18:31:28.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:28.517+0000 7f5110df4640 1 -- 172.21.15.155:0/2422086335 >> 172.21.15.155:0/2422086335 conn(0x7f510c0b7ea0 msgr2=0x7f510c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:28.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:28.517+0000 7f5110df4640 1 -- 172.21.15.155:0/2422086335 shutdown_connections 2024-03-25T18:31:28.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:28.517+0000 7f5110df4640 1 -- 172.21.15.155:0/2422086335 wait complete. 2024-03-25T18:31:28.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:28.517+0000 7f5110df4640 1 librados: shutdown 2024-03-25T18:31:28.530 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:31:35.337 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:31:35.453 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.451+0000 7fda0564e640 10 monclient: get_monmap_and_config 2024-03-25T18:31:35.453 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.451+0000 7fda0564e640 10 monclient: build_initial_monmap 2024-03-25T18:31:35.453 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.451+0000 7fda0564e640 10 monclient: monmap: 2024-03-25T18:31:35.453 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:31:35.453 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:31:35.453 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:31:35.452804+0000 2024-03-25T18:31:35.453 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:31:35.452804+0000 2024-03-25T18:31:35.453 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:35.453 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:31:35.453 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:31:35.454 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:31:35.454 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:31:35.454 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:31:35.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.452+0000 7fda0564e640 1 Processor -- start 2024-03-25T18:31:35.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.452+0000 7fda0564e640 1 -- start start 2024-03-25T18:31:35.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.452+0000 7fda0564e640 10 monclient: init 2024-03-25T18:31:35.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.453+0000 7fda0564e640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:35.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.453+0000 7fda0564e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:31:35.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.453+0000 7fda0564e640 10 monclient(hunting): picked mon.noname-a con 0x7fda000a98f0 addr v1:172.21.15.155:6789/0 2024-03-25T18:31:35.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.453+0000 7fda0564e640 10 monclient(hunting): picked mon.noname-c con 0x7fda0013f710 addr v1:172.21.15.167:6789/0 2024-03-25T18:31:35.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.453+0000 7fda0564e640 10 monclient(hunting): picked mon.noname-b con 0x7fda001430c0 addr v1:172.21.15.155:6790/0 2024-03-25T18:31:35.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.453+0000 7fda0564e640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda00142520 con 0x7fda000a98f0 2024-03-25T18:31:35.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.453+0000 7fda0564e640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda000b28a0 con 0x7fda0013f710 2024-03-25T18:31:35.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.453+0000 7fda0564e640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda000b2a30 con 0x7fda001430c0 2024-03-25T18:31:35.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.453+0000 7fda0564e640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:35.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.453+0000 7fda0564e640 10 monclient(hunting): authenticate will time out at 7941.763374s 2024-03-25T18:31:35.458 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.457+0000 7fd9feffd640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fda000a98f0 0x7fda000a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:53834/0 (socket says 172.21.15.155:53834) 2024-03-25T18:31:35.458 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.457+0000 7fd9feffd640 1 -- 172.21.15.155:0/3865339349 learned_addr learned my addr 172.21.15.155:0/3865339349 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:31:35.472 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.471+0000 7fd9fdffb640 1 -- 172.21.15.155:0/3865339349 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 909587969 0 0) 0x7fda00142520 con 0x7fda000a98f0 2024-03-25T18:31:35.473 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.471+0000 7fd9fdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:31:35.473 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.471+0000 7fd9fdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:35.473 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.471+0000 7fd9fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:35.473 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.471+0000 7fd9fdffb640 10 monclient(hunting): my global_id is 4530 2024-03-25T18:31:35.473 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.471+0000 7fd9fdffb640 1 -- 172.21.15.155:0/3865339349 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9ec003520 con 0x7fda000a98f0 2024-03-25T18:31:35.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.473+0000 7fd9fdffb640 1 -- 172.21.15.155:0/3865339349 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3278739444 0 0) 0x7fda000b2a30 con 0x7fda001430c0 2024-03-25T18:31:35.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.473+0000 7fd9fdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:31:35.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.473+0000 7fd9fdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:35.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.473+0000 7fd9fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:35.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.473+0000 7fd9fdffb640 10 monclient(hunting): my global_id is 4538 2024-03-25T18:31:35.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.473+0000 7fd9fdffb640 1 -- 172.21.15.155:0/3865339349 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda00142520 con 0x7fda001430c0 2024-03-25T18:31:35.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.480+0000 7fd9fdffb640 1 -- 172.21.15.155:0/3865339349 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2825875863 0 0) 0x7fda00142520 con 0x7fda001430c0 2024-03-25T18:31:35.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.480+0000 7fd9fdffb640 1 -- 172.21.15.155:0/3865339349 >> v1:172.21.15.167:6789/0 conn(0x7fda0013f710 legacy=0x7fda001429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:35.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.480+0000 7fd9fdffb640 1 -- 172.21.15.155:0/3865339349 >> v1:172.21.15.155:6789/0 conn(0x7fda000a98f0 legacy=0x7fda000a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:35.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.480+0000 7fd9fdffb640 10 monclient: _finish_hunting 0 2024-03-25T18:31:35.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.480+0000 7fd9fdffb640 1 monclient: found mon.noname-b 2024-03-25T18:31:35.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.480+0000 7fd9fdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:35.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.480+0000 7fd9fdffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-25T18:31:35.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.480+0000 7fd9fdffb640 1 -- 172.21.15.155:0/3865339349 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda000b44c0 con 0x7fda001430c0 2024-03-25T18:31:35.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.480+0000 7fd9fdffb640 10 monclient: _finish_auth 0 2024-03-25T18:31:35.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.480+0000 7fd9fdffb640 10 monclient: _check_auth_tickets 2024-03-25T18:31:35.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.480+0000 7fd9fdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:35.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.480+0000 7fd9fdffb640 1 -- 172.21.15.155:0/3865339349 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fd9f8001670 con 0x7fda001430c0 2024-03-25T18:31:35.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.480+0000 7fd9fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:35.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.480+0000 7fd9fdffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:31:35.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.480+0000 7fd9fdffb640 10 monclient: dump: 2024-03-25T18:31:35.482 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:31:35.482 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:31:35.482 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:35.482 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:35.483 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:35.483 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:31:35.483 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:31:35.483 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:31:35.483 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:31:35.483 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:31:35.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.480+0000 7fda0564e640 5 monclient: authenticate success, global_id 4538 2024-03-25T18:31:35.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.480+0000 7fda0564e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:35.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.484+0000 7fd9fdffb640 1 -- 172.21.15.155:0/3865339349 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd9f80017d0 con 0x7fda001430c0 2024-03-25T18:31:35.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.484+0000 7fd9fdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:35.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.484+0000 7fd9fdffb640 1 -- 172.21.15.155:0/3865339349 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fd9f8001be0 con 0x7fda001430c0 2024-03-25T18:31:35.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.484+0000 7fd9fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:35.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.484+0000 7fd9fdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:31:35.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.484+0000 7fd9fdffb640 10 monclient: dump: 2024-03-25T18:31:35.485 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:31:35.485 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:31:35.486 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:35.486 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:35.486 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:35.486 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:31:35.486 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:31:35.486 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:31:35.486 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:31:35.486 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:31:35.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.484+0000 7fda0564e640 10 monclient: get_monmap_and_config success 2024-03-25T18:31:35.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.484+0000 7fda0564e640 4 set_mon_vals no callback set 2024-03-25T18:31:35.488 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.484+0000 7fda0564e640 10 monclient: shutdown 2024-03-25T18:31:35.488 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.484+0000 7fda0564e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:35.488 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.484+0000 7fda0564e640 1 -- 172.21.15.155:0/3865339349 >> v1:172.21.15.155:6790/0 conn(0x7fda001430c0 legacy=0x7fda0014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:35.488 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.484+0000 7fda0564e640 1 -- 172.21.15.155:0/3865339349 shutdown_connections 2024-03-25T18:31:35.488 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.485+0000 7fda0564e640 1 -- 172.21.15.155:0/3865339349 >> 172.21.15.155:0/3865339349 conn(0x7fda000b7ea0 msgr2=0x7fda000b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:35.488 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.485+0000 7fda0564e640 1 -- 172.21.15.155:0/3865339349 shutdown_connections 2024-03-25T18:31:35.488 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.485+0000 7fda0564e640 1 -- 172.21.15.155:0/3865339349 wait complete. 2024-03-25T18:31:35.488 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.485+0000 7fda0564e640 10 monclient: build_initial_monmap 2024-03-25T18:31:35.488 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.485+0000 7fda0564e640 10 monclient: monmap: 2024-03-25T18:31:35.488 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:31:35.488 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:31:35.488 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:31:35.489 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:31:35.489 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:35.489 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:31:35.489 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:31:35.489 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:31:35.489 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:31:35.489 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:31:35.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.486+0000 7fda0564e640 1 librados: starting msgr at 2024-03-25T18:31:35.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.486+0000 7fda0564e640 1 librados: starting objecter 2024-03-25T18:31:35.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.486+0000 7fda0564e640 1 Processor -- start 2024-03-25T18:31:35.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.486+0000 7fda0564e640 1 -- start start 2024-03-25T18:31:35.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.486+0000 7fda0564e640 1 librados: setting wanted keys 2024-03-25T18:31:35.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.486+0000 7fda0564e640 1 librados: calling monclient init 2024-03-25T18:31:35.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.486+0000 7fda0564e640 10 monclient: init 2024-03-25T18:31:35.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.487+0000 7fda0564e640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:35.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.487+0000 7fda0564e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:31:35.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.487+0000 7fda0564e640 10 monclient(hunting): picked mon.noname-b con 0x7fda000a98f0 addr v1:172.21.15.167:6789/0 2024-03-25T18:31:35.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.487+0000 7fda0564e640 10 monclient(hunting): picked mon.noname-a con 0x7fda0013f710 addr v1:172.21.15.155:6789/0 2024-03-25T18:31:35.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.487+0000 7fda0564e640 10 monclient(hunting): picked mon.noname-c con 0x7fda001430c0 addr v1:172.21.15.155:6790/0 2024-03-25T18:31:35.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.487+0000 7fda0564e640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda0013aae0 con 0x7fda0013f710 2024-03-25T18:31:35.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.487+0000 7fda0564e640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda0013ad50 con 0x7fda000a98f0 2024-03-25T18:31:35.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.487+0000 7fda0564e640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda0013b030 con 0x7fda001430c0 2024-03-25T18:31:35.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.487+0000 7fda0564e640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:35.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.487+0000 7fda0564e640 10 monclient(hunting): authenticate will time out at 7941.797066s 2024-03-25T18:31:35.499 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.498+0000 7fd9ff7fe640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fda001430c0 0x7fda001e29a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:36940/0 (socket says 172.21.15.155:36940) 2024-03-25T18:31:35.499 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.498+0000 7fd9ff7fe640 1 -- 172.21.15.155:0/2372110075 learned_addr learned my addr 172.21.15.155:0/2372110075 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:31:35.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.500+0000 7fd9f77fe640 10 client.?.objecter ms_handle_connect 0x7fda001430c0 2024-03-25T18:31:35.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.500+0000 7fd9f77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:35.502 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.501+0000 7fd9f77fe640 10 client.?.objecter ms_handle_connect 0x7fda000a98f0 2024-03-25T18:31:35.502 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.501+0000 7fd9f77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:35.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.507+0000 7fd9f77fe640 1 -- 172.21.15.155:0/2372110075 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2168880123 0 0) 0x7fda0013ad50 con 0x7fda000a98f0 2024-03-25T18:31:35.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.507+0000 7fd9f77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:31:35.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.507+0000 7fd9f77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:35.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.507+0000 7fd9f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:35.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.507+0000 7fd9f77fe640 10 monclient(hunting): my global_id is 4528 2024-03-25T18:31:35.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.507+0000 7fd9f77fe640 1 -- 172.21.15.155:0/2372110075 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9e0004eb0 con 0x7fda000a98f0 2024-03-25T18:31:35.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.507+0000 7fd9f77fe640 10 client.?.objecter ms_handle_connect 0x7fda0013f710 2024-03-25T18:31:35.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.507+0000 7fd9f77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:35.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.508+0000 7fd9f77fe640 1 -- 172.21.15.155:0/2372110075 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 424164725 0 0) 0x7fda0013b030 con 0x7fda001430c0 2024-03-25T18:31:35.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.508+0000 7fd9f77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:31:35.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.508+0000 7fd9f77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:35.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.508+0000 7fd9f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:35.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.508+0000 7fd9f77fe640 10 monclient(hunting): my global_id is 4541 2024-03-25T18:31:35.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.508+0000 7fd9f77fe640 1 -- 172.21.15.155:0/2372110075 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda0013ad50 con 0x7fda001430c0 2024-03-25T18:31:35.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.518+0000 7fd9f77fe640 1 -- 172.21.15.155:0/2372110075 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2074513340 0 0) 0x7fda0013aae0 con 0x7fda0013f710 2024-03-25T18:31:35.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.518+0000 7fd9f77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:31:35.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.518+0000 7fd9f77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:35.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.518+0000 7fd9f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:35.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.518+0000 7fd9f77fe640 10 monclient(hunting): my global_id is 4533 2024-03-25T18:31:35.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.518+0000 7fd9f77fe640 1 -- 172.21.15.155:0/2372110075 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda0013b030 con 0x7fda0013f710 2024-03-25T18:31:35.520 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.519+0000 7fd9f77fe640 1 -- 172.21.15.155:0/2372110075 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 567439090 0 0) 0x7fd9e0004eb0 con 0x7fda000a98f0 2024-03-25T18:31:35.520 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.519+0000 7fd9f77fe640 1 -- 172.21.15.155:0/2372110075 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fda0013aae0 con 0x7fda000a98f0 2024-03-25T18:31:35.520 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.519+0000 7fd9f77fe640 1 -- 172.21.15.155:0/2372110075 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fd9f0001cd0 con 0x7fda000a98f0 2024-03-25T18:31:35.521 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.519+0000 7fd9f77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:35.521 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.519+0000 7fd9f77fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:31:35.521 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.519+0000 7fd9f77fe640 10 monclient(hunting): dump: 2024-03-25T18:31:35.521 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:31:35.521 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:31:35.521 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:35.521 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:35.521 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:35.521 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:31:35.521 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:31:35.521 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:31:35.521 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:31:35.521 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:31:35.521 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.519+0000 7fd9f77fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:31:35.529 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.528+0000 7fd9f77fe640 1 -- 172.21.15.155:0/2372110075 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1746020727 0 0) 0x7fda0013ad50 con 0x7fda001430c0 2024-03-25T18:31:35.529 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.528+0000 7fd9f77fe640 1 -- 172.21.15.155:0/2372110075 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd9e0004eb0 con 0x7fda001430c0 2024-03-25T18:31:35.529 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.528+0000 7fd9f77fe640 1 -- 172.21.15.155:0/2372110075 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fd9f8001b60 con 0x7fda001430c0 2024-03-25T18:31:35.529 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.528+0000 7fd9f77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:35.529 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.528+0000 7fd9f77fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:31:35.529 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.528+0000 7fd9f77fe640 10 monclient(hunting): dump: 2024-03-25T18:31:35.529 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:31:35.529 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:31:35.529 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:35.530 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:35.530 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:35.530 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:31:35.530 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:31:35.530 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:31:35.530 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:31:35.530 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:31:35.537 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.536+0000 7fd9f77fe640 1 -- 172.21.15.155:0/2372110075 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 837326826 0 0) 0x7fd9e0004eb0 con 0x7fda001430c0 2024-03-25T18:31:35.537 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.536+0000 7fd9f77fe640 1 -- 172.21.15.155:0/2372110075 >> v1:172.21.15.167:6789/0 conn(0x7fda000a98f0 legacy=0x7fda0013ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:35.537 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.536+0000 7fd9f77fe640 1 -- 172.21.15.155:0/2372110075 >> v1:172.21.15.155:6789/0 conn(0x7fda0013f710 legacy=0x7fda001df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:35.538 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.536+0000 7fd9f77fe640 10 monclient: _finish_hunting 0 2024-03-25T18:31:35.538 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.536+0000 7fd9f77fe640 1 monclient: found mon.c 2024-03-25T18:31:35.538 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.536+0000 7fd9f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:35.538 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.536+0000 7fd9f77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:31:35.538 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.536+0000 7fd9f77fe640 1 -- 172.21.15.155:0/2372110075 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda0013b310 con 0x7fda001430c0 2024-03-25T18:31:35.538 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.536+0000 7fd9f77fe640 10 monclient: _finish_auth 0 2024-03-25T18:31:35.538 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.536+0000 7fd9f77fe640 10 monclient: _check_auth_tickets 2024-03-25T18:31:35.538 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.536+0000 7fd9f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:35.538 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.536+0000 7fda0564e640 5 monclient: authenticate success, global_id 4541 2024-03-25T18:31:35.538 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.536+0000 7fda0564e640 10 monclient: _renew_subs 2024-03-25T18:31:35.538 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.536+0000 7fda0564e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:31:35.538 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.536+0000 7fda0564e640 1 -- 172.21.15.155:0/2372110075 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda0013b5f0 con 0x7fda001430c0 2024-03-25T18:31:35.538 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.536+0000 7fda0564e640 10 client.4541.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:31:35.539 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.536+0000 7fda0564e640 10 monclient: _renew_subs 2024-03-25T18:31:35.539 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.536+0000 7fda0564e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:31:35.539 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.536+0000 7fda0564e640 1 -- 172.21.15.155:0/2372110075 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fda0012f6c0 con 0x7fda001430c0 2024-03-25T18:31:35.539 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.536+0000 7fda0564e640 1 librados: init done 2024-03-25T18:31:35.539 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.537+0000 7fda0564e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:35.539 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.537+0000 7fda0564e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:35.539 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.537+0000 7fda0564e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:31:35.540 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.537+0000 7fda0564e640 1 -- 172.21.15.155:0/2372110075 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda000627b0 con 0x7fda001430c0 2024-03-25T18:31:35.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.844+0000 7fd9f77fe640 1 -- 172.21.15.155:0/2372110075 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd9f80038a0 con 0x7fda001430c0 2024-03-25T18:31:35.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.844+0000 7fd9f77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:35.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.845+0000 7fd9f77fe640 1 -- 172.21.15.155:0/2372110075 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fd9f8003cf0 con 0x7fda001430c0 2024-03-25T18:31:35.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.845+0000 7fd9f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:35.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.845+0000 7fd9f77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:31:35.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.845+0000 7fd9f77fe640 10 monclient: dump: 2024-03-25T18:31:35.847 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:31:35.847 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:31:35.847 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:35.847 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:35.847 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:35.847 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:31:35.847 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:31:35.847 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:31:35.847 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:31:35.847 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:31:35.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.845+0000 7fd9fd7fa640 4 set_mon_vals no callback set 2024-03-25T18:31:35.848 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.845+0000 7fd9f77fe640 1 -- 172.21.15.155:0/2372110075 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7fd9f8003f10 con 0x7fda001430c0 2024-03-25T18:31:35.848 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.846+0000 7fd9feffd640 10 monclient: get_auth_request con 0x7fd9e006dfb0 auth_method 0 2024-03-25T18:31:35.848 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.846+0000 7fd9f77fe640 10 client.4541.objecter ms_handle_connect 0x7fd9e006dfb0 2024-03-25T18:31:35.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.852+0000 7fd9f77fe640 1 -- 172.21.15.155:0/2372110075 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7fd9f8054820 con 0x7fda001430c0 2024-03-25T18:31:35.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.852+0000 7fd9f77fe640 10 client.4541.objecter ms_dispatch 0x7fda00000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:31:35.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.852+0000 7fd9f77fe640 3 client.4541.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:31:35.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.852+0000 7fd9f77fe640 3 client.4541.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:31:35.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.852+0000 7fd9f77fe640 20 client.4541.objecter dump_active .. 0 homeless 2024-03-25T18:31:35.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.852+0000 7fd9f77fe640 1 -- 172.21.15.155:0/2372110075 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd9f8050330 con 0x7fda001430c0 2024-03-25T18:31:35.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.852+0000 7fd9f77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:35.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:35.852+0000 7fd9f77fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:31:36.109 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:36.108+0000 7fda0564e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:36.109 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:36.108+0000 7fda0564e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:36.109 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:36.108+0000 7fda0564e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:31:36.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:36.108+0000 7fda0564e640 1 -- 172.21.15.155:0/2372110075 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fda0013de20 con 0x7fda001430c0 2024-03-25T18:31:36.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:36.135+0000 7fd9f77fe640 1 -- 172.21.15.155:0/2372110075 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7fd9f8052260 con 0x7fda001430c0 2024-03-25T18:31:36.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:36.135+0000 7fd9f77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:36.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:36.135+0000 7fd9f77fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:31:36.137 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:31:36.137 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:31:36.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:36.138+0000 7fda0564e640 10 librados: watch_flush enter 2024-03-25T18:31:36.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:36.138+0000 7fda0564e640 10 librados: watch_flush exit 2024-03-25T18:31:36.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:36.138+0000 7fda0564e640 20 client.4541.objecter shutdown clearing up homeless session... 2024-03-25T18:31:36.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:36.138+0000 7fda0564e640 10 client.4541.objecter successfully canceled tick 2024-03-25T18:31:36.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:36.138+0000 7fda0564e640 1 -- 172.21.15.155:0/2372110075 >> v1:172.21.15.167:6800/2919070503 conn(0x7fd9e006dfb0 legacy=0x7fd9e0070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:36.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:36.138+0000 7fda0564e640 10 monclient: shutdown 2024-03-25T18:31:36.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:36.138+0000 7fda0564e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:36.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:36.138+0000 7fda0564e640 1 -- 172.21.15.155:0/2372110075 >> v1:172.21.15.155:6790/0 conn(0x7fda001430c0 legacy=0x7fda001e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:36.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:36.138+0000 7fda0564e640 1 -- 172.21.15.155:0/2372110075 shutdown_connections 2024-03-25T18:31:36.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:36.138+0000 7fda0564e640 1 -- 172.21.15.155:0/2372110075 >> 172.21.15.155:0/2372110075 conn(0x7fda000b7ea0 msgr2=0x7fda00074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:36.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:36.138+0000 7fda0564e640 1 -- 172.21.15.155:0/2372110075 shutdown_connections 2024-03-25T18:31:36.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:36.138+0000 7fda0564e640 1 -- 172.21.15.155:0/2372110075 wait complete. 2024-03-25T18:31:36.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:36.138+0000 7fda0564e640 1 librados: shutdown 2024-03-25T18:31:36.152 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:31:42.958 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:31:43.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.076+0000 7f313749a640 10 monclient: get_monmap_and_config 2024-03-25T18:31:43.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.076+0000 7f313749a640 10 monclient: build_initial_monmap 2024-03-25T18:31:43.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.076+0000 7f313749a640 10 monclient: monmap: 2024-03-25T18:31:43.077 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:31:43.077 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:31:43.077 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:31:43.077312+0000 2024-03-25T18:31:43.078 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:31:43.077312+0000 2024-03-25T18:31:43.078 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:43.078 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:31:43.078 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:31:43.078 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:31:43.078 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:31:43.078 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:31:43.079 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.077+0000 7f313749a640 1 Processor -- start 2024-03-25T18:31:43.079 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.077+0000 7f313749a640 1 -- start start 2024-03-25T18:31:43.079 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.077+0000 7f313749a640 10 monclient: init 2024-03-25T18:31:43.079 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.078+0000 7f313749a640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:43.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.078+0000 7f313749a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:31:43.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.078+0000 7f313749a640 10 monclient(hunting): picked mon.noname-c con 0x7f31300b4d60 addr v1:172.21.15.167:6789/0 2024-03-25T18:31:43.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.078+0000 7f313749a640 10 monclient(hunting): picked mon.noname-b con 0x7f31300b1730 addr v1:172.21.15.155:6790/0 2024-03-25T18:31:43.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.078+0000 7f313749a640 10 monclient(hunting): picked mon.noname-a con 0x7f313014be90 addr v1:172.21.15.155:6789/0 2024-03-25T18:31:43.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.078+0000 7f313749a640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31300af940 con 0x7f313014be90 2024-03-25T18:31:43.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.078+0000 7f313749a640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31300b0a70 con 0x7f31300b4d60 2024-03-25T18:31:43.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.078+0000 7f313749a640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31300b0d50 con 0x7f31300b1730 2024-03-25T18:31:43.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.078+0000 7f313749a640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:43.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.078+0000 7f313749a640 10 monclient(hunting): authenticate will time out at 7949.387982s 2024-03-25T18:31:43.097 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.096+0000 7f3134a35640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f31300b1730 0x7f31300b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:36520/0 (socket says 172.21.15.155:36520) 2024-03-25T18:31:43.097 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.096+0000 7f3134a35640 1 -- 172.21.15.155:0/1943559447 learned_addr learned my addr 172.21.15.155:0/1943559447 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:31:43.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.119+0000 7f3127fff640 1 -- 172.21.15.155:0/1943559447 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3274910520 0 0) 0x7f31300b0d50 con 0x7f31300b1730 2024-03-25T18:31:43.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.119+0000 7f3127fff640 10 monclient(hunting): _negotiate 2024-03-25T18:31:43.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.119+0000 7f3127fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:43.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.119+0000 7f3127fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:43.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.119+0000 7f3127fff640 10 monclient(hunting): my global_id is 4544 2024-03-25T18:31:43.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.119+0000 7f3127fff640 1 -- 172.21.15.155:0/1943559447 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f311c003520 con 0x7f31300b1730 2024-03-25T18:31:43.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.121+0000 7f3127fff640 1 -- 172.21.15.155:0/1943559447 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3253465533 0 0) 0x7f31300af940 con 0x7f313014be90 2024-03-25T18:31:43.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.121+0000 7f3127fff640 10 monclient(hunting): _negotiate 2024-03-25T18:31:43.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.121+0000 7f3127fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:43.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.121+0000 7f3127fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:43.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.121+0000 7f3127fff640 10 monclient(hunting): my global_id is 4536 2024-03-25T18:31:43.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.121+0000 7f3127fff640 1 -- 172.21.15.155:0/1943559447 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31300b0d50 con 0x7f313014be90 2024-03-25T18:31:43.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.122+0000 7f3127fff640 1 -- 172.21.15.155:0/1943559447 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3227296021 0 0) 0x7f31300b0a70 con 0x7f31300b4d60 2024-03-25T18:31:43.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.122+0000 7f3127fff640 10 monclient(hunting): _negotiate 2024-03-25T18:31:43.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.122+0000 7f3127fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:43.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.122+0000 7f3127fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:43.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.122+0000 7f3127fff640 10 monclient(hunting): my global_id is 4531 2024-03-25T18:31:43.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.122+0000 7f3127fff640 1 -- 172.21.15.155:0/1943559447 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31300af940 con 0x7f31300b4d60 2024-03-25T18:31:43.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.132+0000 7f3127fff640 1 -- 172.21.15.155:0/1943559447 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3200433501 0 0) 0x7f31300b0d50 con 0x7f313014be90 2024-03-25T18:31:43.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.132+0000 7f3127fff640 1 -- 172.21.15.155:0/1943559447 >> v1:172.21.15.155:6790/0 conn(0x7f31300b1730 legacy=0x7f31300b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:43.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.132+0000 7f3127fff640 1 -- 172.21.15.155:0/1943559447 >> v1:172.21.15.167:6789/0 conn(0x7f31300b4d60 legacy=0x7f313013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:43.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.132+0000 7f3127fff640 10 monclient: _finish_hunting 0 2024-03-25T18:31:43.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.132+0000 7f3127fff640 1 monclient: found mon.noname-a 2024-03-25T18:31:43.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.132+0000 7f3127fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:43.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.132+0000 7f3127fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-25T18:31:43.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.132+0000 7f3127fff640 1 -- 172.21.15.155:0/1943559447 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31300b1030 con 0x7f313014be90 2024-03-25T18:31:43.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.132+0000 7f3127fff640 10 monclient: _finish_auth 0 2024-03-25T18:31:43.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.132+0000 7f3127fff640 10 monclient: _check_auth_tickets 2024-03-25T18:31:43.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.132+0000 7f3127fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:43.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.132+0000 7f3127fff640 1 -- 172.21.15.155:0/1943559447 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f31280019f0 con 0x7f313014be90 2024-03-25T18:31:43.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.132+0000 7f313749a640 5 monclient: authenticate success, global_id 4536 2024-03-25T18:31:43.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.132+0000 7f313749a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:43.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.133+0000 7f3127fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:43.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.133+0000 7f3127fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:31:43.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.133+0000 7f3127fff640 10 monclient: dump: 2024-03-25T18:31:43.135 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:31:43.135 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:31:43.135 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:43.136 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:43.136 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:43.136 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:31:43.136 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:31:43.136 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:31:43.136 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:31:43.136 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:31:43.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.133+0000 7f313749a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:43.151 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.150+0000 7f3127fff640 1 -- 172.21.15.155:0/1943559447 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3128001b50 con 0x7f313014be90 2024-03-25T18:31:43.151 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.150+0000 7f3127fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:43.151 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.150+0000 7f3127fff640 1 -- 172.21.15.155:0/1943559447 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f3128001f60 con 0x7f313014be90 2024-03-25T18:31:43.151 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.150+0000 7f3127fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:43.152 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.150+0000 7f3127fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:31:43.152 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.150+0000 7f3127fff640 10 monclient: dump: 2024-03-25T18:31:43.152 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:31:43.152 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:31:43.152 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:43.152 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:43.152 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:43.152 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:31:43.152 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:31:43.152 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:31:43.152 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:31:43.152 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:31:43.152 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.150+0000 7f313749a640 10 monclient: get_monmap_and_config success 2024-03-25T18:31:43.152 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.150+0000 7f313749a640 4 set_mon_vals no callback set 2024-03-25T18:31:43.154 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.151+0000 7f313749a640 10 monclient: shutdown 2024-03-25T18:31:43.154 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.151+0000 7f313749a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:43.154 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.151+0000 7f313749a640 1 -- 172.21.15.155:0/1943559447 >> v1:172.21.15.155:6789/0 conn(0x7f313014be90 legacy=0x7f313014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:43.154 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.151+0000 7f313749a640 1 -- 172.21.15.155:0/1943559447 shutdown_connections 2024-03-25T18:31:43.154 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.151+0000 7f313749a640 1 -- 172.21.15.155:0/1943559447 >> 172.21.15.155:0/1943559447 conn(0x7f31300617a0 msgr2=0x7f3130061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:43.154 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.151+0000 7f313749a640 1 -- 172.21.15.155:0/1943559447 shutdown_connections 2024-03-25T18:31:43.154 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.151+0000 7f313749a640 1 -- 172.21.15.155:0/1943559447 wait complete. 2024-03-25T18:31:43.154 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.152+0000 7f313749a640 10 monclient: build_initial_monmap 2024-03-25T18:31:43.155 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.152+0000 7f313749a640 10 monclient: monmap: 2024-03-25T18:31:43.155 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:31:43.155 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:31:43.155 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:31:43.155 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:31:43.155 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:43.155 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:31:43.155 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:31:43.155 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:31:43.155 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:31:43.155 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:31:43.155 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.152+0000 7f313749a640 1 librados: starting msgr at 2024-03-25T18:31:43.155 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.152+0000 7f313749a640 1 librados: starting objecter 2024-03-25T18:31:43.155 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.153+0000 7f313749a640 1 Processor -- start 2024-03-25T18:31:43.155 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.153+0000 7f313749a640 1 -- start start 2024-03-25T18:31:43.155 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.153+0000 7f313749a640 1 librados: setting wanted keys 2024-03-25T18:31:43.156 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.153+0000 7f313749a640 1 librados: calling monclient init 2024-03-25T18:31:43.156 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.153+0000 7f313749a640 10 monclient: init 2024-03-25T18:31:43.156 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.153+0000 7f313749a640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:43.156 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.153+0000 7f313749a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:31:43.157 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.153+0000 7f313749a640 10 monclient(hunting): picked mon.noname-a con 0x7f31300b1730 addr v1:172.21.15.155:6789/0 2024-03-25T18:31:43.157 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.153+0000 7f313749a640 10 monclient(hunting): picked mon.noname-c con 0x7f31300b4d60 addr v1:172.21.15.155:6790/0 2024-03-25T18:31:43.157 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.153+0000 7f313749a640 10 monclient(hunting): picked mon.noname-b con 0x7f313014be90 addr v1:172.21.15.167:6789/0 2024-03-25T18:31:43.157 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.153+0000 7f313749a640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31300aabb0 con 0x7f31300b1730 2024-03-25T18:31:43.157 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.153+0000 7f313749a640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31300a9900 con 0x7f313014be90 2024-03-25T18:31:43.157 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.153+0000 7f313749a640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31300a9470 con 0x7f31300b4d60 2024-03-25T18:31:43.157 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.153+0000 7f313749a640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:43.157 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.154+0000 7f313749a640 10 monclient(hunting): authenticate will time out at 7949.463721s 2024-03-25T18:31:43.165 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.164+0000 7f3134a35640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f31300b4d60 0x7f3130136a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:36568/0 (socket says 172.21.15.155:36568) 2024-03-25T18:31:43.165 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.164+0000 7f3134a35640 1 -- 172.21.15.155:0/3531375724 learned_addr learned my addr 172.21.15.155:0/3531375724 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:31:43.166 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.165+0000 7f3125ffb640 10 client.?.objecter ms_handle_connect 0x7f31300b4d60 2024-03-25T18:31:43.166 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.165+0000 7f3125ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:43.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.171+0000 7f3125ffb640 10 client.?.objecter ms_handle_connect 0x7f313014be90 2024-03-25T18:31:43.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.171+0000 7f3125ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:43.178 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.177+0000 7f3125ffb640 10 client.?.objecter ms_handle_connect 0x7f31300b1730 2024-03-25T18:31:43.178 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.177+0000 7f3125ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:43.181 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.180+0000 7f3125ffb640 1 -- 172.21.15.155:0/3531375724 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 62907955 0 0) 0x7f31300a9900 con 0x7f313014be90 2024-03-25T18:31:43.181 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.180+0000 7f3125ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:31:43.182 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.180+0000 7f3125ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:43.182 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.180+0000 7f3125ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:43.182 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.180+0000 7f3125ffb640 10 monclient(hunting): my global_id is 4534 2024-03-25T18:31:43.182 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.180+0000 7f3125ffb640 1 -- 172.21.15.155:0/3531375724 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3114004eb0 con 0x7f313014be90 2024-03-25T18:31:43.184 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.183+0000 7f3125ffb640 1 -- 172.21.15.155:0/3531375724 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1276327237 0 0) 0x7f31300a9470 con 0x7f31300b4d60 2024-03-25T18:31:43.184 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.183+0000 7f3125ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:31:43.184 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.183+0000 7f3125ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:43.185 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.183+0000 7f3125ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:43.185 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.183+0000 7f3125ffb640 10 monclient(hunting): my global_id is 4547 2024-03-25T18:31:43.185 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.183+0000 7f3125ffb640 1 -- 172.21.15.155:0/3531375724 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31300a9900 con 0x7f31300b4d60 2024-03-25T18:31:43.186 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.185+0000 7f3125ffb640 1 -- 172.21.15.155:0/3531375724 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4049392076 0 0) 0x7f31300aabb0 con 0x7f31300b1730 2024-03-25T18:31:43.186 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.185+0000 7f3125ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:31:43.186 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.185+0000 7f3125ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:43.186 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.185+0000 7f3125ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:43.186 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.185+0000 7f3125ffb640 10 monclient(hunting): my global_id is 4539 2024-03-25T18:31:43.187 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.185+0000 7f3125ffb640 1 -- 172.21.15.155:0/3531375724 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31300a9470 con 0x7f31300b1730 2024-03-25T18:31:43.193 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.192+0000 7f3125ffb640 1 -- 172.21.15.155:0/3531375724 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3378708672 0 0) 0x7f3114004eb0 con 0x7f313014be90 2024-03-25T18:31:43.193 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.192+0000 7f3125ffb640 1 -- 172.21.15.155:0/3531375724 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f31300aabb0 con 0x7f313014be90 2024-03-25T18:31:43.193 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.192+0000 7f3125ffb640 1 -- 172.21.15.155:0/3531375724 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f3128001ea0 con 0x7f313014be90 2024-03-25T18:31:43.193 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.192+0000 7f3125ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:43.193 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.192+0000 7f3125ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:31:43.193 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.192+0000 7f3125ffb640 10 monclient(hunting): dump: 2024-03-25T18:31:43.194 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:31:43.194 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:31:43.194 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:43.194 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:43.194 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:43.194 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:31:43.194 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:31:43.194 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:31:43.194 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:31:43.194 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:31:43.194 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.192+0000 7f3125ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:31:43.205 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.204+0000 7f3125ffb640 1 -- 172.21.15.155:0/3531375724 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4011373291 0 0) 0x7f31300a9470 con 0x7f31300b1730 2024-03-25T18:31:43.205 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.204+0000 7f3125ffb640 1 -- 172.21.15.155:0/3531375724 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3114004eb0 con 0x7f31300b1730 2024-03-25T18:31:43.205 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.204+0000 7f3125ffb640 1 -- 172.21.15.155:0/3531375724 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f312c001830 con 0x7f31300b1730 2024-03-25T18:31:43.205 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.204+0000 7f3125ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:43.205 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.204+0000 7f3125ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:31:43.205 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.204+0000 7f3125ffb640 10 monclient(hunting): dump: 2024-03-25T18:31:43.206 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:31:43.206 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:31:43.206 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:43.206 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:43.206 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:43.206 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:31:43.206 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:31:43.206 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:31:43.206 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:31:43.206 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:31:43.221 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.219+0000 7f3125ffb640 1 -- 172.21.15.155:0/3531375724 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 708403073 0 0) 0x7f3114004eb0 con 0x7f31300b1730 2024-03-25T18:31:43.221 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.219+0000 7f3125ffb640 1 -- 172.21.15.155:0/3531375724 >> v1:172.21.15.155:6790/0 conn(0x7f31300b4d60 legacy=0x7f3130136a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:43.221 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.219+0000 7f3125ffb640 1 -- 172.21.15.155:0/3531375724 >> v1:172.21.15.167:6789/0 conn(0x7f313014be90 legacy=0x7f3130137120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:43.221 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.219+0000 7f3125ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:31:43.221 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.219+0000 7f3125ffb640 1 monclient: found mon.a 2024-03-25T18:31:43.221 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.219+0000 7f3125ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:43.221 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.219+0000 7f3125ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:31:43.221 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.219+0000 7f3125ffb640 1 -- 172.21.15.155:0/3531375724 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31300a9600 con 0x7f31300b1730 2024-03-25T18:31:43.221 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.219+0000 7f3125ffb640 10 monclient: _finish_auth 0 2024-03-25T18:31:43.221 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.219+0000 7f3125ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:31:43.221 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.219+0000 7f3125ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:43.221 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.219+0000 7f313749a640 5 monclient: authenticate success, global_id 4539 2024-03-25T18:31:43.221 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.219+0000 7f313749a640 10 monclient: _renew_subs 2024-03-25T18:31:43.222 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.219+0000 7f313749a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:31:43.222 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.219+0000 7f313749a640 1 -- 172.21.15.155:0/3531375724 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3130137830 con 0x7f31300b1730 2024-03-25T18:31:43.222 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.219+0000 7f313749a640 10 client.4539.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:31:43.222 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.219+0000 7f313749a640 10 monclient: _renew_subs 2024-03-25T18:31:43.222 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.219+0000 7f313749a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:31:43.222 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.219+0000 7f313749a640 1 -- 172.21.15.155:0/3531375724 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3130143610 con 0x7f31300b1730 2024-03-25T18:31:43.222 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.220+0000 7f313749a640 1 librados: init done 2024-03-25T18:31:43.222 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.221+0000 7f313749a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:43.223 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.221+0000 7f313749a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:43.223 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.221+0000 7f313749a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:31:43.223 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.221+0000 7f313749a640 1 -- 172.21.15.155:0/3531375724 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3130061c30 con 0x7f31300b1730 2024-03-25T18:31:43.473 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.472+0000 7f3125ffb640 1 -- 172.21.15.155:0/3531375724 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f312c001d40 con 0x7f31300b1730 2024-03-25T18:31:43.473 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.472+0000 7f3125ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:43.473 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.472+0000 7f3125ffb640 1 -- 172.21.15.155:0/3531375724 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f312c0037b0 con 0x7f31300b1730 2024-03-25T18:31:43.473 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.472+0000 7f3125ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:43.473 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.472+0000 7f3125ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:31:43.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.472+0000 7f3125ffb640 10 monclient: dump: 2024-03-25T18:31:43.474 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:31:43.474 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:31:43.474 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:43.474 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:43.474 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:43.474 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:31:43.474 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:31:43.474 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:31:43.474 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:31:43.474 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:31:43.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.472+0000 7f31277fe640 4 set_mon_vals no callback set 2024-03-25T18:31:43.475 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.472+0000 7f3125ffb640 1 -- 172.21.15.155:0/3531375724 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7f312c019800 con 0x7f31300b1730 2024-03-25T18:31:43.475 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.473+0000 7f3134a35640 10 monclient: get_auth_request con 0x7f311406dfb0 auth_method 0 2024-03-25T18:31:43.475 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.473+0000 7f3125ffb640 10 client.4539.objecter ms_handle_connect 0x7f311406dfb0 2024-03-25T18:31:43.480 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.478+0000 7f3125ffb640 1 -- 172.21.15.155:0/3531375724 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7f312c053580 con 0x7f31300b1730 2024-03-25T18:31:43.480 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.478+0000 7f3125ffb640 10 client.4539.objecter ms_dispatch 0x7f3130000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:31:43.480 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.478+0000 7f3125ffb640 3 client.4539.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:31:43.480 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.478+0000 7f3125ffb640 3 client.4539.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:31:43.480 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.478+0000 7f3125ffb640 20 client.4539.objecter dump_active .. 0 homeless 2024-03-25T18:31:43.480 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.478+0000 7f3125ffb640 1 -- 172.21.15.155:0/3531375724 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f312c04eec0 con 0x7f31300b1730 2024-03-25T18:31:43.480 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.479+0000 7f3125ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:43.480 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.479+0000 7f3125ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:31:43.699 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.698+0000 7f313749a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:43.699 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.698+0000 7f313749a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:43.699 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.698+0000 7f313749a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:31:43.699 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.698+0000 7f313749a640 1 -- 172.21.15.155:0/3531375724 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f31300627b0 con 0x7f31300b1730 2024-03-25T18:31:43.758 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.757+0000 7f3125ffb640 1 -- 172.21.15.155:0/3531375724 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7f312c050fc0 con 0x7f31300b1730 2024-03-25T18:31:43.758 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.757+0000 7f3125ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:43.758 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.757+0000 7f3125ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:31:43.758 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:31:43.759 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:31:43.761 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.759+0000 7f313749a640 10 librados: watch_flush enter 2024-03-25T18:31:43.761 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.759+0000 7f313749a640 10 librados: watch_flush exit 2024-03-25T18:31:43.761 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.759+0000 7f313749a640 20 client.4539.objecter shutdown clearing up homeless session... 2024-03-25T18:31:43.761 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.759+0000 7f313749a640 10 client.4539.objecter successfully canceled tick 2024-03-25T18:31:43.761 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.759+0000 7f313749a640 1 -- 172.21.15.155:0/3531375724 >> v1:172.21.15.167:6800/2919070503 conn(0x7f311406dfb0 legacy=0x7f3114070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:43.761 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.759+0000 7f313749a640 10 monclient: shutdown 2024-03-25T18:31:43.761 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.759+0000 7f313749a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:43.761 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.759+0000 7f313749a640 1 -- 172.21.15.155:0/3531375724 >> v1:172.21.15.155:6789/0 conn(0x7f31300b1730 legacy=0x7f3130140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:43.761 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.759+0000 7f313749a640 1 -- 172.21.15.155:0/3531375724 shutdown_connections 2024-03-25T18:31:43.761 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.759+0000 7f313749a640 1 -- 172.21.15.155:0/3531375724 >> 172.21.15.155:0/3531375724 conn(0x7f31300617a0 msgr2=0x7f313013c0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:43.761 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.759+0000 7f313749a640 1 -- 172.21.15.155:0/3531375724 shutdown_connections 2024-03-25T18:31:43.761 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.759+0000 7f313749a640 1 -- 172.21.15.155:0/3531375724 wait complete. 2024-03-25T18:31:43.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:43.760+0000 7f313749a640 1 librados: shutdown 2024-03-25T18:31:43.774 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:31:46.722 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:07:20.326 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:46.729 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:07:20.334 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:46.770 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:07:20.374 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:46.782 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:07:20.361 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:46.789 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:07:20.393 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:46.789 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:07:20.367 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:46.794 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:07:20.399 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:46.812 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:07:20.390 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:46.814 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:07:20.419 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:50.580 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:31:50.697 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.695+0000 7eff096bb640 10 monclient: get_monmap_and_config 2024-03-25T18:31:50.697 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.695+0000 7eff096bb640 10 monclient: build_initial_monmap 2024-03-25T18:31:50.698 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.695+0000 7eff096bb640 10 monclient: monmap: 2024-03-25T18:31:50.698 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:31:50.698 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:31:50.698 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:31:50.697521+0000 2024-03-25T18:31:50.698 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:31:50.697521+0000 2024-03-25T18:31:50.698 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:50.698 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:31:50.698 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:31:50.698 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:31:50.698 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:31:50.698 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:31:50.699 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.696+0000 7eff096bb640 1 Processor -- start 2024-03-25T18:31:50.699 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.696+0000 7eff096bb640 1 -- start start 2024-03-25T18:31:50.699 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.696+0000 7eff096bb640 10 monclient: init 2024-03-25T18:31:50.699 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.696+0000 7eff096bb640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:50.700 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.696+0000 7eff096bb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:31:50.700 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.696+0000 7eff096bb640 10 monclient(hunting): picked mon.noname-b con 0x7eff040a98f0 addr v1:172.21.15.155:6790/0 2024-03-25T18:31:50.700 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.696+0000 7eff096bb640 10 monclient(hunting): picked mon.noname-a con 0x7eff0413f710 addr v1:172.21.15.155:6789/0 2024-03-25T18:31:50.700 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.696+0000 7eff096bb640 10 monclient(hunting): picked mon.noname-c con 0x7eff041430c0 addr v1:172.21.15.167:6789/0 2024-03-25T18:31:50.700 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.696+0000 7eff096bb640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff04142520 con 0x7eff0413f710 2024-03-25T18:31:50.700 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.696+0000 7eff096bb640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff040b28a0 con 0x7eff041430c0 2024-03-25T18:31:50.700 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.696+0000 7eff096bb640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff040b2a30 con 0x7eff040a98f0 2024-03-25T18:31:50.700 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.696+0000 7eff096bb640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:50.700 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.696+0000 7eff096bb640 10 monclient(hunting): authenticate will time out at 7957.007624s 2024-03-25T18:31:50.704 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.702+0000 7eff02ffd640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7eff040a98f0 0x7eff040a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:49300/0 (socket says 172.21.15.155:49300) 2024-03-25T18:31:50.704 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.702+0000 7eff02ffd640 1 -- 172.21.15.155:0/4117838279 learned_addr learned my addr 172.21.15.155:0/4117838279 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:31:50.717 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.715+0000 7eff01ffb640 1 -- 172.21.15.155:0/4117838279 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2408515964 0 0) 0x7eff04142520 con 0x7eff0413f710 2024-03-25T18:31:50.717 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.715+0000 7eff01ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:31:50.717 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.715+0000 7eff01ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:50.717 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.715+0000 7eff01ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:50.717 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.715+0000 7eff01ffb640 10 monclient(hunting): my global_id is 4542 2024-03-25T18:31:50.717 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.715+0000 7eff01ffb640 1 -- 172.21.15.155:0/4117838279 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efeec003520 con 0x7eff0413f710 2024-03-25T18:31:50.721 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.718+0000 7eff01ffb640 1 -- 172.21.15.155:0/4117838279 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 805844018 0 0) 0x7eff040b2a30 con 0x7eff040a98f0 2024-03-25T18:31:50.721 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.718+0000 7eff01ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:31:50.721 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.718+0000 7eff01ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:50.721 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.718+0000 7eff01ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:50.721 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.718+0000 7eff01ffb640 10 monclient(hunting): my global_id is 4550 2024-03-25T18:31:50.721 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.718+0000 7eff01ffb640 1 -- 172.21.15.155:0/4117838279 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff04142520 con 0x7eff040a98f0 2024-03-25T18:31:50.735 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.732+0000 7eff01ffb640 1 -- 172.21.15.155:0/4117838279 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3156267640 0 0) 0x7efeec003520 con 0x7eff0413f710 2024-03-25T18:31:50.735 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.732+0000 7eff01ffb640 1 -- 172.21.15.155:0/4117838279 >> v1:172.21.15.155:6790/0 conn(0x7eff040a98f0 legacy=0x7eff040a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:50.735 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.732+0000 7eff01ffb640 1 -- 172.21.15.155:0/4117838279 >> v1:172.21.15.167:6789/0 conn(0x7eff041430c0 legacy=0x7eff0414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:50.735 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.732+0000 7eff01ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:31:50.735 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.732+0000 7eff01ffb640 1 monclient: found mon.noname-a 2024-03-25T18:31:50.735 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.732+0000 7eff01ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:50.735 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.732+0000 7eff01ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-25T18:31:50.735 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.732+0000 7eff01ffb640 1 -- 172.21.15.155:0/4117838279 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff040b44c0 con 0x7eff0413f710 2024-03-25T18:31:50.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.732+0000 7eff01ffb640 10 monclient: _finish_auth 0 2024-03-25T18:31:50.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.732+0000 7eff01ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:31:50.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.732+0000 7eff01ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:50.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.732+0000 7eff01ffb640 1 -- 172.21.15.155:0/4117838279 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7efef8001690 con 0x7eff0413f710 2024-03-25T18:31:50.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.732+0000 7eff01ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:50.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.732+0000 7eff01ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:31:50.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.732+0000 7eff01ffb640 10 monclient: dump: 2024-03-25T18:31:50.736 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:31:50.736 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:31:50.736 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:50.736 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:50.736 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:50.736 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:31:50.736 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:31:50.736 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:31:50.737 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:31:50.737 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:31:50.737 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.732+0000 7eff096bb640 5 monclient: authenticate success, global_id 4542 2024-03-25T18:31:50.737 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.733+0000 7eff096bb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:50.750 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.748+0000 7eff01ffb640 1 -- 172.21.15.155:0/4117838279 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efef80017f0 con 0x7eff0413f710 2024-03-25T18:31:50.750 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.748+0000 7eff01ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:50.750 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.748+0000 7eff01ffb640 1 -- 172.21.15.155:0/4117838279 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7efef8001c00 con 0x7eff0413f710 2024-03-25T18:31:50.750 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.748+0000 7eff096bb640 10 monclient: get_monmap_and_config success 2024-03-25T18:31:50.750 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.748+0000 7eff096bb640 4 set_mon_vals no callback set 2024-03-25T18:31:50.750 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.748+0000 7eff01ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:50.750 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.748+0000 7eff01ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:31:50.750 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.748+0000 7eff01ffb640 10 monclient: dump: 2024-03-25T18:31:50.750 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:31:50.751 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:31:50.751 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:50.751 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:50.751 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:50.751 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:31:50.751 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:31:50.751 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:31:50.751 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:31:50.751 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:31:50.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.748+0000 7eff096bb640 10 monclient: shutdown 2024-03-25T18:31:50.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.748+0000 7eff096bb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:50.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.748+0000 7eff096bb640 1 -- 172.21.15.155:0/4117838279 >> v1:172.21.15.155:6789/0 conn(0x7eff0413f710 legacy=0x7eff041429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:50.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.749+0000 7eff096bb640 1 -- 172.21.15.155:0/4117838279 shutdown_connections 2024-03-25T18:31:50.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.749+0000 7eff096bb640 1 -- 172.21.15.155:0/4117838279 >> 172.21.15.155:0/4117838279 conn(0x7eff040b7ea0 msgr2=0x7eff040b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:50.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.749+0000 7eff096bb640 1 -- 172.21.15.155:0/4117838279 shutdown_connections 2024-03-25T18:31:50.753 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.749+0000 7eff096bb640 1 -- 172.21.15.155:0/4117838279 wait complete. 2024-03-25T18:31:50.753 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.749+0000 7eff096bb640 10 monclient: build_initial_monmap 2024-03-25T18:31:50.753 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.749+0000 7eff096bb640 10 monclient: monmap: 2024-03-25T18:31:50.753 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:31:50.753 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:31:50.753 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:31:50.753 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:31:50.753 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:50.753 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:31:50.753 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:31:50.753 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:31:50.753 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:31:50.753 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:31:50.753 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.750+0000 7eff096bb640 1 librados: starting msgr at 2024-03-25T18:31:50.753 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.750+0000 7eff096bb640 1 librados: starting objecter 2024-03-25T18:31:50.753 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.750+0000 7eff096bb640 1 Processor -- start 2024-03-25T18:31:50.753 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.750+0000 7eff096bb640 1 -- start start 2024-03-25T18:31:50.754 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.750+0000 7eff096bb640 1 librados: setting wanted keys 2024-03-25T18:31:50.754 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.750+0000 7eff096bb640 1 librados: calling monclient init 2024-03-25T18:31:50.754 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.750+0000 7eff096bb640 10 monclient: init 2024-03-25T18:31:50.755 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.750+0000 7eff096bb640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:50.755 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.750+0000 7eff096bb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:31:50.755 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.750+0000 7eff096bb640 10 monclient(hunting): picked mon.noname-a con 0x7eff040a98f0 addr v1:172.21.15.155:6789/0 2024-03-25T18:31:50.755 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.751+0000 7eff096bb640 10 monclient(hunting): picked mon.noname-b con 0x7eff0413f710 addr v1:172.21.15.167:6789/0 2024-03-25T18:31:50.755 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.751+0000 7eff096bb640 10 monclient(hunting): picked mon.noname-c con 0x7eff041430c0 addr v1:172.21.15.155:6790/0 2024-03-25T18:31:50.755 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.751+0000 7eff096bb640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff041397c0 con 0x7eff040a98f0 2024-03-25T18:31:50.755 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.751+0000 7eff096bb640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff041daa20 con 0x7eff0413f710 2024-03-25T18:31:50.755 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.751+0000 7eff096bb640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff041d8210 con 0x7eff041430c0 2024-03-25T18:31:50.755 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.751+0000 7eff096bb640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:50.755 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.751+0000 7eff096bb640 10 monclient(hunting): authenticate will time out at 7957.061820s 2024-03-25T18:31:50.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.755+0000 7eff02ffd640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7eff040a98f0 0x7eff041daf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:57588/0 (socket says 172.21.15.155:57588) 2024-03-25T18:31:50.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.755+0000 7eff02ffd640 1 -- 172.21.15.155:0/2661113610 learned_addr learned my addr 172.21.15.155:0/2661113610 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:31:50.759 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.757+0000 7efeeb7fe640 10 client.?.objecter ms_handle_connect 0x7eff040a98f0 2024-03-25T18:31:50.760 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.757+0000 7efeeb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:50.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.767+0000 7efeeb7fe640 10 client.?.objecter ms_handle_connect 0x7eff041430c0 2024-03-25T18:31:50.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.767+0000 7efeeb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:50.771 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.769+0000 7efeeb7fe640 10 client.?.objecter ms_handle_connect 0x7eff0413f710 2024-03-25T18:31:50.771 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.769+0000 7efeeb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:50.775 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.773+0000 7efeeb7fe640 1 -- 172.21.15.155:0/2661113610 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1267133668 0 0) 0x7eff041397c0 con 0x7eff040a98f0 2024-03-25T18:31:50.775 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.773+0000 7efeeb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:31:50.775 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.773+0000 7efeeb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:50.775 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.773+0000 7efeeb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:50.775 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.773+0000 7efeeb7fe640 10 monclient(hunting): my global_id is 4545 2024-03-25T18:31:50.775 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.773+0000 7efeeb7fe640 1 -- 172.21.15.155:0/2661113610 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efef0004eb0 con 0x7eff040a98f0 2024-03-25T18:31:50.781 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.778+0000 7efeeb7fe640 1 -- 172.21.15.155:0/2661113610 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4210092091 0 0) 0x7eff041d8210 con 0x7eff041430c0 2024-03-25T18:31:50.781 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.778+0000 7efeeb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:31:50.781 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.778+0000 7efeeb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:50.781 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.778+0000 7efeeb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:50.781 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.778+0000 7efeeb7fe640 10 monclient(hunting): my global_id is 4553 2024-03-25T18:31:50.781 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.778+0000 7efeeb7fe640 1 -- 172.21.15.155:0/2661113610 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff041397c0 con 0x7eff041430c0 2024-03-25T18:31:50.785 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.783+0000 7efeeb7fe640 1 -- 172.21.15.155:0/2661113610 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 325601253 0 0) 0x7eff041daa20 con 0x7eff0413f710 2024-03-25T18:31:50.785 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.783+0000 7efeeb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:31:50.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.783+0000 7efeeb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:50.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.783+0000 7efeeb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:50.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.783+0000 7efeeb7fe640 10 monclient(hunting): my global_id is 4540 2024-03-25T18:31:50.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.783+0000 7efeeb7fe640 1 -- 172.21.15.155:0/2661113610 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff041d8210 con 0x7eff0413f710 2024-03-25T18:31:50.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.784+0000 7efeeb7fe640 1 -- 172.21.15.155:0/2661113610 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 487808870 0 0) 0x7efef0004eb0 con 0x7eff040a98f0 2024-03-25T18:31:50.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.784+0000 7efeeb7fe640 1 -- 172.21.15.155:0/2661113610 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7eff041daa20 con 0x7eff040a98f0 2024-03-25T18:31:50.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.784+0000 7efeeb7fe640 1 -- 172.21.15.155:0/2661113610 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7efef40019b0 con 0x7eff040a98f0 2024-03-25T18:31:50.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.784+0000 7efeeb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:50.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.784+0000 7efeeb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:31:50.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.784+0000 7efeeb7fe640 10 monclient(hunting): dump: 2024-03-25T18:31:50.787 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:31:50.787 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:31:50.787 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:50.787 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:50.787 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:50.787 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:31:50.787 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:31:50.787 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:31:50.787 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:31:50.787 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:31:50.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.784+0000 7efeeb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:31:50.792 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.789+0000 7efeeb7fe640 1 -- 172.21.15.155:0/2661113610 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3359019268 0 0) 0x7eff041daa20 con 0x7eff040a98f0 2024-03-25T18:31:50.792 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.789+0000 7efeeb7fe640 1 -- 172.21.15.155:0/2661113610 >> v1:172.21.15.155:6790/0 conn(0x7eff041430c0 legacy=0x7eff041d7b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:50.792 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.789+0000 7efeeb7fe640 1 -- 172.21.15.155:0/2661113610 >> v1:172.21.15.167:6789/0 conn(0x7eff0413f710 legacy=0x7eff041d73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:50.792 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.789+0000 7efeeb7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:31:50.792 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.789+0000 7efeeb7fe640 1 monclient: found mon.a 2024-03-25T18:31:50.792 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.789+0000 7efeeb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:50.793 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.789+0000 7efeeb7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:31:50.793 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.789+0000 7efeeb7fe640 1 -- 172.21.15.155:0/2661113610 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff041db650 con 0x7eff040a98f0 2024-03-25T18:31:50.793 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.789+0000 7efeeb7fe640 10 monclient: _finish_auth 0 2024-03-25T18:31:50.793 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.789+0000 7efeeb7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:31:50.793 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.789+0000 7efeeb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:50.793 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.789+0000 7eff096bb640 5 monclient: authenticate success, global_id 4545 2024-03-25T18:31:50.793 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.789+0000 7eff096bb640 10 monclient: _renew_subs 2024-03-25T18:31:50.793 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.789+0000 7eff096bb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:31:50.793 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.789+0000 7eff096bb640 1 -- 172.21.15.155:0/2661113610 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff04137720 con 0x7eff040a98f0 2024-03-25T18:31:50.793 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.789+0000 7eff096bb640 10 client.4545.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:31:50.793 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.789+0000 7eff096bb640 10 monclient: _renew_subs 2024-03-25T18:31:50.793 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.789+0000 7eff096bb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:31:50.793 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.789+0000 7eff096bb640 1 -- 172.21.15.155:0/2661113610 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7eff04137d50 con 0x7eff040a98f0 2024-03-25T18:31:50.793 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.789+0000 7eff096bb640 1 librados: init done 2024-03-25T18:31:50.794 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.790+0000 7eff096bb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:50.794 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.790+0000 7eff096bb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:50.794 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.790+0000 7eff096bb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:31:50.794 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:50.790+0000 7eff096bb640 1 -- 172.21.15.155:0/2661113610 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff041db840 con 0x7eff040a98f0 2024-03-25T18:31:51.088 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:51.086+0000 7efeeb7fe640 1 -- 172.21.15.155:0/2661113610 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efef40047e0 con 0x7eff040a98f0 2024-03-25T18:31:51.088 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:51.086+0000 7efeeb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:51.088 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:51.086+0000 7efeeb7fe640 1 -- 172.21.15.155:0/2661113610 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7efef4004990 con 0x7eff040a98f0 2024-03-25T18:31:51.088 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:51.086+0000 7efeeb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:51.089 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:51.086+0000 7efeeb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:31:51.089 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:51.086+0000 7efeeb7fe640 10 monclient: dump: 2024-03-25T18:31:51.089 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:31:51.089 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:31:51.089 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:51.089 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:51.089 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:51.089 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:31:51.089 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:31:51.089 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:31:51.089 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:31:51.089 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:31:51.089 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:51.086+0000 7eff017fa640 4 set_mon_vals no callback set 2024-03-25T18:31:51.090 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:51.087+0000 7efeeb7fe640 1 -- 172.21.15.155:0/2661113610 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7efef4004bb0 con 0x7eff040a98f0 2024-03-25T18:31:51.090 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:51.088+0000 7eff027fc640 10 monclient: get_auth_request con 0x7efef006ddf0 auth_method 0 2024-03-25T18:31:51.090 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:51.088+0000 7efeeb7fe640 10 client.4545.objecter ms_handle_connect 0x7efef006ddf0 2024-03-25T18:31:51.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:51.093+0000 7efeeb7fe640 1 -- 172.21.15.155:0/2661113610 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7efef4054700 con 0x7eff040a98f0 2024-03-25T18:31:51.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:51.093+0000 7efeeb7fe640 10 client.4545.objecter ms_dispatch 0x7eff04000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:31:51.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:51.093+0000 7efeeb7fe640 3 client.4545.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:31:51.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:51.093+0000 7efeeb7fe640 3 client.4545.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:31:51.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:51.093+0000 7efeeb7fe640 20 client.4545.objecter dump_active .. 0 homeless 2024-03-25T18:31:51.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:51.093+0000 7efeeb7fe640 1 -- 172.21.15.155:0/2661113610 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efef4050210 con 0x7eff040a98f0 2024-03-25T18:31:51.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:51.093+0000 7efeeb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:51.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:51.093+0000 7efeeb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:31:51.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:51.339+0000 7eff096bb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:51.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:51.339+0000 7eff096bb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:51.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:51.339+0000 7eff096bb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:31:51.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:51.339+0000 7eff096bb640 1 -- 172.21.15.155:0/2661113610 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7eff041dabb0 con 0x7eff040a98f0 2024-03-25T18:31:51.394 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:51.391+0000 7efeeb7fe640 1 -- 172.21.15.155:0/2661113610 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7efef4052140 con 0x7eff040a98f0 2024-03-25T18:31:51.394 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:51.391+0000 7efeeb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:51.394 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:51.391+0000 7efeeb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:31:51.394 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:31:51.394 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:31:51.397 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:51.393+0000 7eff096bb640 10 librados: watch_flush enter 2024-03-25T18:31:51.397 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:51.394+0000 7eff096bb640 10 librados: watch_flush exit 2024-03-25T18:31:51.397 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:51.394+0000 7eff096bb640 20 client.4545.objecter shutdown clearing up homeless session... 2024-03-25T18:31:51.397 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:51.394+0000 7eff096bb640 10 client.4545.objecter successfully canceled tick 2024-03-25T18:31:51.397 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:51.394+0000 7eff096bb640 1 -- 172.21.15.155:0/2661113610 >> v1:172.21.15.167:6800/2919070503 conn(0x7efef006ddf0 legacy=0x7efef00702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:51.397 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:51.394+0000 7eff096bb640 10 monclient: shutdown 2024-03-25T18:31:51.397 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:51.394+0000 7eff096bb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:51.397 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:51.394+0000 7eff096bb640 1 -- 172.21.15.155:0/2661113610 >> v1:172.21.15.155:6789/0 conn(0x7eff040a98f0 legacy=0x7eff041daf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:51.397 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:51.394+0000 7eff096bb640 1 -- 172.21.15.155:0/2661113610 shutdown_connections 2024-03-25T18:31:51.397 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:51.394+0000 7eff096bb640 1 -- 172.21.15.155:0/2661113610 >> 172.21.15.155:0/2661113610 conn(0x7eff040b7ea0 msgr2=0x7eff0413c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:51.397 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:51.394+0000 7eff096bb640 1 -- 172.21.15.155:0/2661113610 shutdown_connections 2024-03-25T18:31:51.397 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:51.394+0000 7eff096bb640 1 -- 172.21.15.155:0/2661113610 wait complete. 2024-03-25T18:31:51.397 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:51.394+0000 7eff096bb640 1 librados: shutdown 2024-03-25T18:31:51.408 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:31:58.215 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:31:58.332 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.330+0000 7f5b3421e640 10 monclient: get_monmap_and_config 2024-03-25T18:31:58.332 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.330+0000 7f5b3421e640 10 monclient: build_initial_monmap 2024-03-25T18:31:58.332 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.330+0000 7f5b3421e640 10 monclient: monmap: 2024-03-25T18:31:58.332 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:31:58.332 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:31:58.332 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:31:58.331599+0000 2024-03-25T18:31:58.332 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:31:58.331599+0000 2024-03-25T18:31:58.332 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:58.332 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:31:58.333 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:31:58.333 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:31:58.333 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:31:58.333 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:31:58.334 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.331+0000 7f5b3421e640 1 Processor -- start 2024-03-25T18:31:58.334 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.331+0000 7f5b3421e640 1 -- start start 2024-03-25T18:31:58.334 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.331+0000 7f5b3421e640 10 monclient: init 2024-03-25T18:31:58.334 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.332+0000 7f5b3421e640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:58.335 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.332+0000 7f5b3421e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:31:58.335 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.332+0000 7f5b3421e640 10 monclient(hunting): picked mon.noname-b con 0x7f5b2c0a98f0 addr v1:172.21.15.155:6790/0 2024-03-25T18:31:58.335 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.332+0000 7f5b3421e640 10 monclient(hunting): picked mon.noname-c con 0x7f5b2c13f710 addr v1:172.21.15.167:6789/0 2024-03-25T18:31:58.335 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.332+0000 7f5b3421e640 10 monclient(hunting): picked mon.noname-a con 0x7f5b2c1430c0 addr v1:172.21.15.155:6789/0 2024-03-25T18:31:58.335 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.332+0000 7f5b3421e640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b2c142520 con 0x7f5b2c1430c0 2024-03-25T18:31:58.335 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.332+0000 7f5b3421e640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b2c0b28a0 con 0x7f5b2c13f710 2024-03-25T18:31:58.335 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.332+0000 7f5b3421e640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b2c0b2a30 con 0x7f5b2c0a98f0 2024-03-25T18:31:58.335 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.332+0000 7f5b3421e640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:58.335 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.332+0000 7f5b3421e640 10 monclient(hunting): authenticate will time out at 7964.642356s 2024-03-25T18:31:58.340 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.339+0000 7f5b317b9640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f5b2c13f710 0x7f5b2c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:34526/0 (socket says 172.21.15.155:34526) 2024-03-25T18:31:58.340 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.339+0000 7f5b317b9640 1 -- 172.21.15.155:0/2773757454 learned_addr learned my addr 172.21.15.155:0/2773757454 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:31:58.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.351+0000 7f5b30fb8640 1 -- 172.21.15.155:0/2773757454 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3274289779 0 0) 0x7f5b2c0b28a0 con 0x7f5b2c13f710 2024-03-25T18:31:58.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.351+0000 7f5b30fb8640 10 monclient(hunting): _negotiate 2024-03-25T18:31:58.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.351+0000 7f5b30fb8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:58.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.351+0000 7f5b30fb8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:58.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.351+0000 7f5b30fb8640 10 monclient(hunting): my global_id is 4543 2024-03-25T18:31:58.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.351+0000 7f5b30fb8640 1 -- 172.21.15.155:0/2773757454 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b18003520 con 0x7f5b2c13f710 2024-03-25T18:31:58.357 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.355+0000 7f5b30fb8640 1 -- 172.21.15.155:0/2773757454 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3858665823 0 0) 0x7f5b2c0b2a30 con 0x7f5b2c0a98f0 2024-03-25T18:31:58.357 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.355+0000 7f5b30fb8640 10 monclient(hunting): _negotiate 2024-03-25T18:31:58.357 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.355+0000 7f5b30fb8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:58.357 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.355+0000 7f5b30fb8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:58.357 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.355+0000 7f5b30fb8640 10 monclient(hunting): my global_id is 4556 2024-03-25T18:31:58.357 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.355+0000 7f5b30fb8640 1 -- 172.21.15.155:0/2773757454 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b2c0b28a0 con 0x7f5b2c0a98f0 2024-03-25T18:31:58.362 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.361+0000 7f5b30fb8640 1 -- 172.21.15.155:0/2773757454 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 286449506 0 0) 0x7f5b18003520 con 0x7f5b2c13f710 2024-03-25T18:31:58.362 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.361+0000 7f5b30fb8640 1 -- 172.21.15.155:0/2773757454 >> v1:172.21.15.155:6790/0 conn(0x7f5b2c0a98f0 legacy=0x7f5b2c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:58.362 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.361+0000 7f5b30fb8640 1 -- 172.21.15.155:0/2773757454 >> v1:172.21.15.155:6789/0 conn(0x7f5b2c1430c0 legacy=0x7f5b2c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:58.362 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.361+0000 7f5b30fb8640 10 monclient: _finish_hunting 0 2024-03-25T18:31:58.363 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.361+0000 7f5b30fb8640 1 monclient: found mon.noname-c 2024-03-25T18:31:58.363 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.361+0000 7f5b30fb8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:58.363 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.361+0000 7f5b30fb8640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.167:6789/0 2024-03-25T18:31:58.363 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.361+0000 7f5b30fb8640 1 -- 172.21.15.155:0/2773757454 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b2c0b44c0 con 0x7f5b2c13f710 2024-03-25T18:31:58.363 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.361+0000 7f5b30fb8640 10 monclient: _finish_auth 0 2024-03-25T18:31:58.363 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.361+0000 7f5b30fb8640 10 monclient: _check_auth_tickets 2024-03-25T18:31:58.363 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.361+0000 7f5b30fb8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:58.363 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.361+0000 7f5b30fb8640 1 -- 172.21.15.155:0/2773757454 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f5b1c001690 con 0x7f5b2c13f710 2024-03-25T18:31:58.363 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.361+0000 7f5b30fb8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:58.363 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.361+0000 7f5b30fb8640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:31:58.363 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.361+0000 7f5b30fb8640 10 monclient: dump: 2024-03-25T18:31:58.363 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:31:58.363 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:31:58.363 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:58.364 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:58.364 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:58.364 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:31:58.364 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:31:58.364 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:31:58.364 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:31:58.364 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:31:58.364 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.361+0000 7f5b3421e640 5 monclient: authenticate success, global_id 4543 2024-03-25T18:31:58.364 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.361+0000 7f5b3421e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:58.365 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.364+0000 7f5b30fb8640 1 -- 172.21.15.155:0/2773757454 <== mon.1 v1:172.21.15.167:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5b1c0017f0 con 0x7f5b2c13f710 2024-03-25T18:31:58.365 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.364+0000 7f5b30fb8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:58.365 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.364+0000 7f5b3421e640 10 monclient: get_monmap_and_config success 2024-03-25T18:31:58.365 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.364+0000 7f5b3421e640 4 set_mon_vals no callback set 2024-03-25T18:31:58.366 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.365+0000 7f5b3421e640 10 monclient: shutdown 2024-03-25T18:31:58.367 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.365+0000 7f5b3421e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:58.367 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.365+0000 7f5b3421e640 1 -- 172.21.15.155:0/2773757454 >> v1:172.21.15.167:6789/0 conn(0x7f5b2c13f710 legacy=0x7f5b2c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:58.367 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.365+0000 7f5b3421e640 1 -- 172.21.15.155:0/2773757454 shutdown_connections 2024-03-25T18:31:58.367 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.365+0000 7f5b3421e640 1 -- 172.21.15.155:0/2773757454 >> 172.21.15.155:0/2773757454 conn(0x7f5b2c0b7ea0 msgr2=0x7f5b2c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:58.367 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.365+0000 7f5b3421e640 1 -- 172.21.15.155:0/2773757454 shutdown_connections 2024-03-25T18:31:58.367 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.365+0000 7f5b3421e640 1 -- 172.21.15.155:0/2773757454 wait complete. 2024-03-25T18:31:58.369 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.366+0000 7f5b3421e640 10 monclient: build_initial_monmap 2024-03-25T18:31:58.369 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.366+0000 7f5b3421e640 10 monclient: monmap: 2024-03-25T18:31:58.369 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:31:58.369 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:31:58.369 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:31:58.369 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:31:58.369 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:58.369 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:31:58.369 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:31:58.369 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:31:58.370 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:31:58.370 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:31:58.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.366+0000 7f5b3421e640 1 librados: starting msgr at 2024-03-25T18:31:58.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.366+0000 7f5b3421e640 1 librados: starting objecter 2024-03-25T18:31:58.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.367+0000 7f5b3421e640 1 Processor -- start 2024-03-25T18:31:58.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.367+0000 7f5b3421e640 1 -- start start 2024-03-25T18:31:58.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.367+0000 7f5b3421e640 1 librados: setting wanted keys 2024-03-25T18:31:58.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.367+0000 7f5b3421e640 1 librados: calling monclient init 2024-03-25T18:31:58.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.367+0000 7f5b3421e640 10 monclient: init 2024-03-25T18:31:58.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.367+0000 7f5b3421e640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:58.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.367+0000 7f5b3421e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:31:58.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.367+0000 7f5b3421e640 10 monclient(hunting): picked mon.noname-a con 0x7f5b2c0a98f0 addr v1:172.21.15.155:6789/0 2024-03-25T18:31:58.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.367+0000 7f5b3421e640 10 monclient(hunting): picked mon.noname-c con 0x7f5b2c13f710 addr v1:172.21.15.155:6790/0 2024-03-25T18:31:58.371 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.367+0000 7f5b3421e640 10 monclient(hunting): picked mon.noname-b con 0x7f5b2c1430c0 addr v1:172.21.15.167:6789/0 2024-03-25T18:31:58.371 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.367+0000 7f5b3421e640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b2c136a10 con 0x7f5b2c0a98f0 2024-03-25T18:31:58.371 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.367+0000 7f5b3421e640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b2c133b60 con 0x7f5b2c1430c0 2024-03-25T18:31:58.371 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.367+0000 7f5b3421e640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b2c0ab950 con 0x7f5b2c13f710 2024-03-25T18:31:58.371 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.367+0000 7f5b3421e640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:58.371 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.367+0000 7f5b3421e640 10 monclient(hunting): authenticate will time out at 7964.677395s 2024-03-25T18:31:58.383 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.382+0000 7f5b317b9640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f5b2c13f710 0x7f5b2c136300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:50454/0 (socket says 172.21.15.155:50454) 2024-03-25T18:31:58.383 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.382+0000 7f5b317b9640 1 -- 172.21.15.155:0/2597025874 learned_addr learned my addr 172.21.15.155:0/2597025874 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:31:58.385 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.384+0000 7f5b227fc640 10 client.?.objecter ms_handle_connect 0x7f5b2c13f710 2024-03-25T18:31:58.385 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.384+0000 7f5b227fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:58.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.387+0000 7f5b227fc640 10 client.?.objecter ms_handle_connect 0x7f5b2c1430c0 2024-03-25T18:31:58.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.387+0000 7f5b227fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:58.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.388+0000 7f5b227fc640 10 client.?.objecter ms_handle_connect 0x7f5b2c0a98f0 2024-03-25T18:31:58.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.388+0000 7f5b227fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:58.396 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.395+0000 7f5b227fc640 1 -- 172.21.15.155:0/2597025874 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 322293855 0 0) 0x7f5b2c136a10 con 0x7f5b2c0a98f0 2024-03-25T18:31:58.396 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.395+0000 7f5b227fc640 10 monclient(hunting): _negotiate 2024-03-25T18:31:58.396 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.395+0000 7f5b227fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:58.396 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.395+0000 7f5b227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:58.396 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.395+0000 7f5b227fc640 10 monclient(hunting): my global_id is 4551 2024-03-25T18:31:58.396 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.395+0000 7f5b227fc640 1 -- 172.21.15.155:0/2597025874 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b10004eb0 con 0x7f5b2c0a98f0 2024-03-25T18:31:58.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.396+0000 7f5b227fc640 1 -- 172.21.15.155:0/2597025874 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4182907551 0 0) 0x7f5b2c0ab950 con 0x7f5b2c13f710 2024-03-25T18:31:58.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.396+0000 7f5b227fc640 10 monclient(hunting): _negotiate 2024-03-25T18:31:58.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.396+0000 7f5b227fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:58.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.396+0000 7f5b227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:58.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.396+0000 7f5b227fc640 10 monclient(hunting): my global_id is 4559 2024-03-25T18:31:58.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.396+0000 7f5b227fc640 1 -- 172.21.15.155:0/2597025874 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b2c136a10 con 0x7f5b2c13f710 2024-03-25T18:31:58.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.397+0000 7f5b227fc640 1 -- 172.21.15.155:0/2597025874 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4241232516 0 0) 0x7f5b2c133b60 con 0x7f5b2c1430c0 2024-03-25T18:31:58.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.397+0000 7f5b227fc640 10 monclient(hunting): _negotiate 2024-03-25T18:31:58.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.397+0000 7f5b227fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:58.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.397+0000 7f5b227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:58.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.397+0000 7f5b227fc640 10 monclient(hunting): my global_id is 4546 2024-03-25T18:31:58.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.397+0000 7f5b227fc640 1 -- 172.21.15.155:0/2597025874 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b2c0ab950 con 0x7f5b2c1430c0 2024-03-25T18:31:58.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.407+0000 7f5b227fc640 1 -- 172.21.15.155:0/2597025874 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3309566476 0 0) 0x7f5b2c0ab950 con 0x7f5b2c1430c0 2024-03-25T18:31:58.409 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.407+0000 7f5b227fc640 1 -- 172.21.15.155:0/2597025874 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b2c133b60 con 0x7f5b2c1430c0 2024-03-25T18:31:58.409 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.407+0000 7f5b227fc640 1 -- 172.21.15.155:0/2597025874 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f5b24001a60 con 0x7f5b2c1430c0 2024-03-25T18:31:58.409 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.407+0000 7f5b227fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:58.409 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.407+0000 7f5b227fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:31:58.409 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.407+0000 7f5b227fc640 10 monclient(hunting): dump: 2024-03-25T18:31:58.409 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:31:58.409 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:31:58.409 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:58.409 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:58.409 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:58.409 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:31:58.409 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:31:58.409 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:31:58.410 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:31:58.410 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:31:58.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.407+0000 7f5b227fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:31:58.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.408+0000 7f5b227fc640 1 -- 172.21.15.155:0/2597025874 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2837358604 0 0) 0x7f5b2c136a10 con 0x7f5b2c13f710 2024-03-25T18:31:58.411 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.408+0000 7f5b227fc640 1 -- 172.21.15.155:0/2597025874 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b2c0ab950 con 0x7f5b2c13f710 2024-03-25T18:31:58.411 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.408+0000 7f5b227fc640 1 -- 172.21.15.155:0/2597025874 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f5b1c001ce0 con 0x7f5b2c13f710 2024-03-25T18:31:58.411 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.408+0000 7f5b227fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:58.411 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.408+0000 7f5b227fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:31:58.411 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.408+0000 7f5b227fc640 10 monclient(hunting): dump: 2024-03-25T18:31:58.411 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:31:58.411 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:31:58.411 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:58.411 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:58.411 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:58.411 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:31:58.411 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:31:58.411 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:31:58.411 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:31:58.411 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:31:58.416 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.415+0000 7f5b227fc640 1 -- 172.21.15.155:0/2597025874 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4026244650 0 0) 0x7f5b10004eb0 con 0x7f5b2c0a98f0 2024-03-25T18:31:58.416 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.415+0000 7f5b227fc640 1 -- 172.21.15.155:0/2597025874 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b2c136a10 con 0x7f5b2c0a98f0 2024-03-25T18:31:58.416 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.415+0000 7f5b227fc640 1 -- 172.21.15.155:0/2597025874 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f5b28001930 con 0x7f5b2c0a98f0 2024-03-25T18:31:58.416 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.415+0000 7f5b227fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:58.416 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.415+0000 7f5b227fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:31:58.417 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.415+0000 7f5b227fc640 10 monclient(hunting): dump: 2024-03-25T18:31:58.417 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:31:58.417 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:31:58.417 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:58.417 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:58.417 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:58.417 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:31:58.417 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:31:58.417 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:31:58.417 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:31:58.417 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:31:58.424 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.422+0000 7f5b227fc640 1 -- 172.21.15.155:0/2597025874 <== mon.1 v1:172.21.15.167:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1533821618 0 0) 0x7f5b2c133b60 con 0x7f5b2c1430c0 2024-03-25T18:31:58.424 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.422+0000 7f5b227fc640 1 -- 172.21.15.155:0/2597025874 >> v1:172.21.15.155:6790/0 conn(0x7f5b2c13f710 legacy=0x7f5b2c136300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:58.424 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.422+0000 7f5b227fc640 1 -- 172.21.15.155:0/2597025874 >> v1:172.21.15.155:6789/0 conn(0x7f5b2c0a98f0 legacy=0x7f5b2c132d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:58.424 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.422+0000 7f5b227fc640 10 monclient: _finish_hunting 0 2024-03-25T18:31:58.424 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.422+0000 7f5b227fc640 1 monclient: found mon.b 2024-03-25T18:31:58.424 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.422+0000 7f5b227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:58.424 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.422+0000 7f5b227fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:31:58.424 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.422+0000 7f5b227fc640 1 -- 172.21.15.155:0/2597025874 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b2c0abc30 con 0x7f5b2c1430c0 2024-03-25T18:31:58.424 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.422+0000 7f5b227fc640 10 monclient: _finish_auth 0 2024-03-25T18:31:58.424 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.422+0000 7f5b227fc640 10 monclient: _check_auth_tickets 2024-03-25T18:31:58.424 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.422+0000 7f5b227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:58.424 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.422+0000 7f5b3421e640 5 monclient: authenticate success, global_id 4546 2024-03-25T18:31:58.425 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.422+0000 7f5b3421e640 10 monclient: _renew_subs 2024-03-25T18:31:58.425 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.422+0000 7f5b3421e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:31:58.425 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.422+0000 7f5b3421e640 1 -- 172.21.15.155:0/2597025874 --> v1:172.21.15.167:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b2c0abf10 con 0x7f5b2c1430c0 2024-03-25T18:31:58.425 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.422+0000 7f5b3421e640 10 client.4546.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:31:58.425 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.422+0000 7f5b3421e640 10 monclient: _renew_subs 2024-03-25T18:31:58.425 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.422+0000 7f5b3421e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:31:58.425 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.422+0000 7f5b3421e640 1 -- 172.21.15.155:0/2597025874 --> v1:172.21.15.167:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b2c137e00 con 0x7f5b2c1430c0 2024-03-25T18:31:58.425 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.422+0000 7f5b3421e640 1 librados: init done 2024-03-25T18:31:58.426 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.423+0000 7f5b3421e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:58.426 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.423+0000 7f5b3421e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:58.426 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.423+0000 7f5b3421e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:31:58.426 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.423+0000 7f5b3421e640 1 -- 172.21.15.155:0/2597025874 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b2c139020 con 0x7f5b2c1430c0 2024-03-25T18:31:58.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.722+0000 7f5b227fc640 1 -- 172.21.15.155:0/2597025874 <== mon.1 v1:172.21.15.167:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5b24001f90 con 0x7f5b2c1430c0 2024-03-25T18:31:58.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.722+0000 7f5b227fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:58.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.722+0000 7f5b227fc640 1 -- 172.21.15.155:0/2597025874 <== mon.1 v1:172.21.15.167:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f5b24003b10 con 0x7f5b2c1430c0 2024-03-25T18:31:58.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.722+0000 7f5b227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:58.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.722+0000 7f5b227fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:31:58.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.722+0000 7f5b227fc640 10 monclient: dump: 2024-03-25T18:31:58.724 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:31:58.724 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:31:58.724 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:58.725 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:31:58.725 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:58.725 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:31:58.725 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:31:58.725 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:31:58.725 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:31:58.725 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:31:58.725 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.722+0000 7f5b23fff640 4 set_mon_vals no callback set 2024-03-25T18:31:58.725 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.723+0000 7f5b227fc640 1 -- 172.21.15.155:0/2597025874 <== mon.1 v1:172.21.15.167:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7f5b24019ae0 con 0x7f5b2c1430c0 2024-03-25T18:31:58.726 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.724+0000 7f5b31fba640 10 monclient: get_auth_request con 0x7f5b1006e0f0 auth_method 0 2024-03-25T18:31:58.726 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.724+0000 7f5b227fc640 10 client.4546.objecter ms_handle_connect 0x7f5b1006e0f0 2024-03-25T18:31:58.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.728+0000 7f5b227fc640 1 -- 172.21.15.155:0/2597025874 <== mon.1 v1:172.21.15.167:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7f5b240536d0 con 0x7f5b2c1430c0 2024-03-25T18:31:58.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.729+0000 7f5b227fc640 10 client.4546.objecter ms_dispatch 0x7f5b2c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:31:58.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.729+0000 7f5b227fc640 3 client.4546.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:31:58.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.729+0000 7f5b227fc640 3 client.4546.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:31:58.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.729+0000 7f5b227fc640 20 client.4546.objecter dump_active .. 0 homeless 2024-03-25T18:31:58.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.729+0000 7f5b227fc640 1 -- 172.21.15.155:0/2597025874 <== mon.1 v1:172.21.15.167:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5b2404f1e0 con 0x7f5b2c1430c0 2024-03-25T18:31:58.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.729+0000 7f5b227fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:58.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.729+0000 7f5b227fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:31:58.984 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.983+0000 7f5b3421e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:58.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.983+0000 7f5b3421e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:58.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.983+0000 7f5b3421e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:31:58.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:58.983+0000 7f5b3421e640 1 -- 172.21.15.155:0/2597025874 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5b2c0627b0 con 0x7f5b2c1430c0 2024-03-25T18:31:59.019 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:59.017+0000 7f5b227fc640 1 -- 172.21.15.155:0/2597025874 <== mon.1 v1:172.21.15.167:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7f5b24051110 con 0x7f5b2c1430c0 2024-03-25T18:31:59.019 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:59.017+0000 7f5b227fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:59.019 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:59.017+0000 7f5b227fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:31:59.019 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:31:59.019 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:31:59.022 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:59.020+0000 7f5b3421e640 10 librados: watch_flush enter 2024-03-25T18:31:59.022 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:59.020+0000 7f5b3421e640 10 librados: watch_flush exit 2024-03-25T18:31:59.022 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:59.020+0000 7f5b3421e640 20 client.4546.objecter shutdown clearing up homeless session... 2024-03-25T18:31:59.022 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:59.020+0000 7f5b3421e640 10 client.4546.objecter successfully canceled tick 2024-03-25T18:31:59.022 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:59.020+0000 7f5b3421e640 1 -- 172.21.15.155:0/2597025874 >> v1:172.21.15.167:6800/2919070503 conn(0x7f5b1006e0f0 legacy=0x7f5b100705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:59.022 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:59.020+0000 7f5b3421e640 10 monclient: shutdown 2024-03-25T18:31:59.022 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:59.020+0000 7f5b3421e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:59.022 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:59.020+0000 7f5b3421e640 1 -- 172.21.15.155:0/2597025874 >> v1:172.21.15.167:6789/0 conn(0x7f5b2c1430c0 legacy=0x7f5b2c133450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:59.022 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:59.020+0000 7f5b3421e640 1 -- 172.21.15.155:0/2597025874 shutdown_connections 2024-03-25T18:31:59.022 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:59.020+0000 7f5b3421e640 1 -- 172.21.15.155:0/2597025874 >> 172.21.15.155:0/2597025874 conn(0x7f5b2c0b7ea0 msgr2=0x7f5b2c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:59.022 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:59.020+0000 7f5b3421e640 1 -- 172.21.15.155:0/2597025874 shutdown_connections 2024-03-25T18:31:59.023 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:59.020+0000 7f5b3421e640 1 -- 172.21.15.155:0/2597025874 wait complete. 2024-03-25T18:31:59.023 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:31:59.020+0000 7f5b3421e640 1 librados: shutdown 2024-03-25T18:31:59.034 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:32:05.841 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:32:05.958 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.956+0000 7f853a54d640 10 monclient: get_monmap_and_config 2024-03-25T18:32:05.958 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.956+0000 7f853a54d640 10 monclient: build_initial_monmap 2024-03-25T18:32:05.958 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.956+0000 7f853a54d640 10 monclient: monmap: 2024-03-25T18:32:05.958 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:32:05.958 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:32:05.958 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:32:05.957448+0000 2024-03-25T18:32:05.958 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:32:05.957448+0000 2024-03-25T18:32:05.958 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:05.959 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:32:05.959 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:32:05.959 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:32:05.959 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:32:05.959 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:32:05.960 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.957+0000 7f853a54d640 1 Processor -- start 2024-03-25T18:32:05.960 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.957+0000 7f853a54d640 1 -- start start 2024-03-25T18:32:05.960 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.957+0000 7f853a54d640 10 monclient: init 2024-03-25T18:32:05.960 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.957+0000 7f853a54d640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:05.961 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.957+0000 7f853a54d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:32:05.961 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.958+0000 7f853a54d640 10 monclient(hunting): picked mon.noname-a con 0x7f85340b4d60 addr v1:172.21.15.155:6789/0 2024-03-25T18:32:05.961 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.958+0000 7f853a54d640 10 monclient(hunting): picked mon.noname-b con 0x7f85340b1730 addr v1:172.21.15.155:6790/0 2024-03-25T18:32:05.961 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.958+0000 7f853a54d640 10 monclient(hunting): picked mon.noname-c con 0x7f853414be90 addr v1:172.21.15.167:6789/0 2024-03-25T18:32:05.961 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.958+0000 7f853a54d640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85340af940 con 0x7f85340b4d60 2024-03-25T18:32:05.961 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.958+0000 7f853a54d640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85340b0a70 con 0x7f853414be90 2024-03-25T18:32:05.961 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.958+0000 7f853a54d640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85340b0d50 con 0x7f85340b1730 2024-03-25T18:32:05.961 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.958+0000 7f853a54d640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:05.961 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.958+0000 7f853a54d640 10 monclient(hunting): authenticate will time out at 7972.267811s 2024-03-25T18:32:05.963 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.962+0000 7f85337fe640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f85340b1730 0x7f85340b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:55218/0 (socket says 172.21.15.155:55218) 2024-03-25T18:32:05.963 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.962+0000 7f85337fe640 1 -- 172.21.15.155:0/3576704117 learned_addr learned my addr 172.21.15.155:0/3576704117 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:32:05.974 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.973+0000 7f8532ffd640 1 -- 172.21.15.155:0/3576704117 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 6526568 0 0) 0x7f85340b0d50 con 0x7f85340b1730 2024-03-25T18:32:05.974 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.973+0000 7f8532ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:32:05.974 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.973+0000 7f8532ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:05.974 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.973+0000 7f8532ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:05.974 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.973+0000 7f8532ffd640 10 monclient(hunting): my global_id is 4562 2024-03-25T18:32:05.975 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.973+0000 7f8532ffd640 1 -- 172.21.15.155:0/3576704117 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f851c003520 con 0x7f85340b1730 2024-03-25T18:32:05.975 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.974+0000 7f8532ffd640 1 -- 172.21.15.155:0/3576704117 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 412476978 0 0) 0x7f85340b0a70 con 0x7f853414be90 2024-03-25T18:32:05.975 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.974+0000 7f8532ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:32:05.975 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.974+0000 7f8532ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:05.976 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.974+0000 7f8532ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:05.976 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.974+0000 7f8532ffd640 10 monclient(hunting): my global_id is 4549 2024-03-25T18:32:05.976 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.974+0000 7f8532ffd640 1 -- 172.21.15.155:0/3576704117 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85340b0d50 con 0x7f853414be90 2024-03-25T18:32:05.981 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.979+0000 7f8532ffd640 1 -- 172.21.15.155:0/3576704117 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1283753716 0 0) 0x7f85340af940 con 0x7f85340b4d60 2024-03-25T18:32:05.981 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.979+0000 7f8532ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:32:05.981 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.979+0000 7f8532ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:05.981 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.979+0000 7f8532ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:05.981 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.979+0000 7f8532ffd640 10 monclient(hunting): my global_id is 4554 2024-03-25T18:32:05.981 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.979+0000 7f8532ffd640 1 -- 172.21.15.155:0/3576704117 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85340b0a70 con 0x7f85340b4d60 2024-03-25T18:32:05.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.989+0000 7f8532ffd640 1 -- 172.21.15.155:0/3576704117 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3888068267 0 0) 0x7f851c003520 con 0x7f85340b1730 2024-03-25T18:32:05.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.989+0000 7f8532ffd640 1 -- 172.21.15.155:0/3576704117 >> v1:172.21.15.167:6789/0 conn(0x7f853414be90 legacy=0x7f853414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:05.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.989+0000 7f8532ffd640 1 -- 172.21.15.155:0/3576704117 >> v1:172.21.15.155:6789/0 conn(0x7f85340b4d60 legacy=0x7f853413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:05.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.989+0000 7f8532ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:32:05.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.989+0000 7f8532ffd640 1 monclient: found mon.noname-b 2024-03-25T18:32:05.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.989+0000 7f8532ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:05.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.989+0000 7f8532ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-25T18:32:05.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.989+0000 7f8532ffd640 1 -- 172.21.15.155:0/3576704117 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85340b1030 con 0x7f85340b1730 2024-03-25T18:32:05.991 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.989+0000 7f8532ffd640 10 monclient: _finish_auth 0 2024-03-25T18:32:05.991 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.989+0000 7f8532ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:32:05.991 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.989+0000 7f8532ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:05.991 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.989+0000 7f8532ffd640 1 -- 172.21.15.155:0/3576704117 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f8528001690 con 0x7f85340b1730 2024-03-25T18:32:05.991 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.989+0000 7f853a54d640 5 monclient: authenticate success, global_id 4562 2024-03-25T18:32:05.991 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.989+0000 7f853a54d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:05.991 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.989+0000 7f8532ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:05.991 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.989+0000 7f8532ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:32:05.991 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.989+0000 7f8532ffd640 10 monclient: dump: 2024-03-25T18:32:05.991 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:32:05.991 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:32:05.991 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:05.991 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:05.991 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:05.991 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:32:05.991 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:32:05.992 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:32:05.992 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:32:05.992 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:32:05.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.989+0000 7f853a54d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:05.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.995+0000 7f8532ffd640 1 -- 172.21.15.155:0/3576704117 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f85280017f0 con 0x7f85340b1730 2024-03-25T18:32:05.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.995+0000 7f8532ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:05.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.995+0000 7f8532ffd640 1 -- 172.21.15.155:0/3576704117 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f8528001c00 con 0x7f85340b1730 2024-03-25T18:32:05.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.995+0000 7f853a54d640 10 monclient: get_monmap_and_config success 2024-03-25T18:32:05.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.995+0000 7f853a54d640 4 set_mon_vals no callback set 2024-03-25T18:32:05.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.995+0000 7f8532ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:05.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.995+0000 7f8532ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:32:05.997 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.995+0000 7f8532ffd640 10 monclient: dump: 2024-03-25T18:32:05.997 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:32:05.997 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:32:05.997 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:05.997 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:05.997 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:05.997 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:32:05.997 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:32:05.997 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:32:05.997 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:32:05.997 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:32:05.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.995+0000 7f853a54d640 10 monclient: shutdown 2024-03-25T18:32:05.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.995+0000 7f853a54d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:05.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.995+0000 7f853a54d640 1 -- 172.21.15.155:0/3576704117 >> v1:172.21.15.155:6790/0 conn(0x7f85340b1730 legacy=0x7f85340b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:05.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.996+0000 7f853a54d640 1 -- 172.21.15.155:0/3576704117 shutdown_connections 2024-03-25T18:32:05.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.996+0000 7f853a54d640 1 -- 172.21.15.155:0/3576704117 >> 172.21.15.155:0/3576704117 conn(0x7f85340617a0 msgr2=0x7f8534061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:05.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.996+0000 7f853a54d640 1 -- 172.21.15.155:0/3576704117 shutdown_connections 2024-03-25T18:32:05.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.996+0000 7f853a54d640 1 -- 172.21.15.155:0/3576704117 wait complete. 2024-03-25T18:32:05.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.997+0000 7f853a54d640 10 monclient: build_initial_monmap 2024-03-25T18:32:05.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.997+0000 7f853a54d640 10 monclient: monmap: 2024-03-25T18:32:05.999 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:32:05.999 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:32:05.999 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:32:05.999 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:32:05.999 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:05.999 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:32:06.000 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:32:06.000 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:32:06.000 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:32:06.000 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:32:06.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.997+0000 7f853a54d640 1 librados: starting msgr at 2024-03-25T18:32:06.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.997+0000 7f853a54d640 1 librados: starting objecter 2024-03-25T18:32:06.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.997+0000 7f853a54d640 1 Processor -- start 2024-03-25T18:32:06.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.997+0000 7f853a54d640 1 -- start start 2024-03-25T18:32:06.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.997+0000 7f853a54d640 1 librados: setting wanted keys 2024-03-25T18:32:06.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.997+0000 7f853a54d640 1 librados: calling monclient init 2024-03-25T18:32:06.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.997+0000 7f853a54d640 10 monclient: init 2024-03-25T18:32:06.001 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.998+0000 7f853a54d640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:06.001 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.998+0000 7f853a54d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:32:06.001 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.998+0000 7f853a54d640 10 monclient(hunting): picked mon.noname-b con 0x7f85340b1730 addr v1:172.21.15.167:6789/0 2024-03-25T18:32:06.001 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.998+0000 7f853a54d640 10 monclient(hunting): picked mon.noname-c con 0x7f85340b4d60 addr v1:172.21.15.155:6790/0 2024-03-25T18:32:06.001 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.998+0000 7f853a54d640 10 monclient(hunting): picked mon.noname-a con 0x7f853414be90 addr v1:172.21.15.155:6789/0 2024-03-25T18:32:06.001 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.998+0000 7f853a54d640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8534145470 con 0x7f853414be90 2024-03-25T18:32:06.001 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.998+0000 7f853a54d640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8534145900 con 0x7f85340b1730 2024-03-25T18:32:06.002 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.998+0000 7f853a54d640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8534136af0 con 0x7f85340b4d60 2024-03-25T18:32:06.002 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.998+0000 7f853a54d640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:06.002 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:05.998+0000 7f853a54d640 10 monclient(hunting): authenticate will time out at 7972.308093s 2024-03-25T18:32:06.013 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.012+0000 7f85337fe640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f85340b4d60 0x7f853412ee50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:55234/0 (socket says 172.21.15.155:55234) 2024-03-25T18:32:06.013 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.012+0000 7f85337fe640 1 -- 172.21.15.155:0/833238670 learned_addr learned my addr 172.21.15.155:0/833238670 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:32:06.015 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.014+0000 7f8530ff9640 10 client.?.objecter ms_handle_connect 0x7f85340b4d60 2024-03-25T18:32:06.015 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.014+0000 7f8530ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:06.016 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.015+0000 7f8530ff9640 10 client.?.objecter ms_handle_connect 0x7f853414be90 2024-03-25T18:32:06.016 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.015+0000 7f8530ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:06.017 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.016+0000 7f8530ff9640 10 client.?.objecter ms_handle_connect 0x7f85340b1730 2024-03-25T18:32:06.017 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.016+0000 7f8530ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:06.029 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.028+0000 7f8530ff9640 1 -- 172.21.15.155:0/833238670 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3920248744 0 0) 0x7f8534136af0 con 0x7f85340b4d60 2024-03-25T18:32:06.030 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.028+0000 7f8530ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:32:06.030 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.028+0000 7f8530ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:06.030 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.028+0000 7f8530ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:06.030 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.028+0000 7f8530ff9640 10 monclient(hunting): my global_id is 4565 2024-03-25T18:32:06.030 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.028+0000 7f8530ff9640 1 -- 172.21.15.155:0/833238670 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8520004eb0 con 0x7f85340b4d60 2024-03-25T18:32:06.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.033+0000 7f8530ff9640 1 -- 172.21.15.155:0/833238670 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1034048142 0 0) 0x7f8534145900 con 0x7f85340b1730 2024-03-25T18:32:06.035 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.033+0000 7f8530ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:32:06.035 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.033+0000 7f8530ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:06.035 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.033+0000 7f8530ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:06.035 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.033+0000 7f8530ff9640 10 monclient(hunting): my global_id is 4552 2024-03-25T18:32:06.035 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.033+0000 7f8530ff9640 1 -- 172.21.15.155:0/833238670 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8534136af0 con 0x7f85340b1730 2024-03-25T18:32:06.037 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.036+0000 7f8530ff9640 1 -- 172.21.15.155:0/833238670 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2975563873 0 0) 0x7f8534145470 con 0x7f853414be90 2024-03-25T18:32:06.037 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.036+0000 7f8530ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:32:06.037 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.036+0000 7f8530ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:06.037 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.036+0000 7f8530ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:06.037 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.036+0000 7f8530ff9640 10 monclient(hunting): my global_id is 4557 2024-03-25T18:32:06.037 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.036+0000 7f8530ff9640 1 -- 172.21.15.155:0/833238670 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8534145900 con 0x7f853414be90 2024-03-25T18:32:06.041 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.039+0000 7f8530ff9640 1 -- 172.21.15.155:0/833238670 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 260565586 0 0) 0x7f8520004eb0 con 0x7f85340b4d60 2024-03-25T18:32:06.041 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.039+0000 7f8530ff9640 1 -- 172.21.15.155:0/833238670 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8534145470 con 0x7f85340b4d60 2024-03-25T18:32:06.041 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.039+0000 7f8530ff9640 1 -- 172.21.15.155:0/833238670 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f8528001bc0 con 0x7f85340b4d60 2024-03-25T18:32:06.041 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.039+0000 7f8530ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:06.041 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.039+0000 7f8530ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:32:06.041 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.039+0000 7f8530ff9640 10 monclient(hunting): dump: 2024-03-25T18:32:06.041 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:32:06.041 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:32:06.041 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:06.041 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:06.041 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:06.041 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:32:06.041 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:32:06.042 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:32:06.042 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:32:06.042 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:32:06.042 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.039+0000 7f8530ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:32:06.045 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.044+0000 7f8530ff9640 1 -- 172.21.15.155:0/833238670 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3941346329 0 0) 0x7f8534136af0 con 0x7f85340b1730 2024-03-25T18:32:06.046 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.044+0000 7f8530ff9640 1 -- 172.21.15.155:0/833238670 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8520004eb0 con 0x7f85340b1730 2024-03-25T18:32:06.046 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.044+0000 7f8530ff9640 1 -- 172.21.15.155:0/833238670 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f8524001870 con 0x7f85340b1730 2024-03-25T18:32:06.046 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.044+0000 7f8530ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:06.046 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.044+0000 7f8530ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:32:06.046 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.044+0000 7f8530ff9640 10 monclient(hunting): dump: 2024-03-25T18:32:06.046 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:32:06.046 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:32:06.046 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:06.046 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:06.046 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:06.046 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:32:06.046 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:32:06.046 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:32:06.046 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:32:06.046 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:32:06.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.059+0000 7f8530ff9640 1 -- 172.21.15.155:0/833238670 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3704513413 0 0) 0x7f8534145470 con 0x7f85340b4d60 2024-03-25T18:32:06.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.059+0000 7f8530ff9640 1 -- 172.21.15.155:0/833238670 >> v1:172.21.15.167:6789/0 conn(0x7f85340b1730 legacy=0x7f85341371f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:06.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.059+0000 7f8530ff9640 1 -- 172.21.15.155:0/833238670 >> v1:172.21.15.155:6789/0 conn(0x7f853414be90 legacy=0x7f85341e36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:06.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.060+0000 7f8530ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:32:06.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.060+0000 7f8530ff9640 1 monclient: found mon.c 2024-03-25T18:32:06.062 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.060+0000 7f8530ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:06.062 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.060+0000 7f8530ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:32:06.062 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.060+0000 7f8530ff9640 1 -- 172.21.15.155:0/833238670 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8534136dd0 con 0x7f85340b4d60 2024-03-25T18:32:06.062 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.060+0000 7f8530ff9640 10 monclient: _finish_auth 0 2024-03-25T18:32:06.062 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.060+0000 7f8530ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:32:06.062 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.060+0000 7f8530ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:06.062 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.060+0000 7f853a54d640 5 monclient: authenticate success, global_id 4565 2024-03-25T18:32:06.062 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.060+0000 7f853a54d640 10 monclient: _renew_subs 2024-03-25T18:32:06.062 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.060+0000 7f853a54d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:32:06.062 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.060+0000 7f853a54d640 1 -- 172.21.15.155:0/833238670 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8534145000 con 0x7f85340b4d60 2024-03-25T18:32:06.062 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.060+0000 7f853a54d640 10 client.4565.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:32:06.062 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.060+0000 7f853a54d640 10 monclient: _renew_subs 2024-03-25T18:32:06.062 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.060+0000 7f853a54d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:32:06.063 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.060+0000 7f853a54d640 1 -- 172.21.15.155:0/833238670 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8534135ce0 con 0x7f85340b4d60 2024-03-25T18:32:06.063 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.060+0000 7f853a54d640 1 librados: init done 2024-03-25T18:32:06.063 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.061+0000 7f853a54d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:06.063 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.061+0000 7f853a54d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:06.063 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.061+0000 7f853a54d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:32:06.063 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.061+0000 7f853a54d640 1 -- 172.21.15.155:0/833238670 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85340afbd0 con 0x7f85340b4d60 2024-03-25T18:32:06.378 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.376+0000 7f8530ff9640 1 -- 172.21.15.155:0/833238670 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f85280011f0 con 0x7f85340b4d60 2024-03-25T18:32:06.378 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.376+0000 7f8530ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:06.378 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.376+0000 7f8530ff9640 1 -- 172.21.15.155:0/833238670 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f8528004b10 con 0x7f85340b4d60 2024-03-25T18:32:06.378 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.376+0000 7f8530ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:06.378 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.376+0000 7f8530ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:32:06.379 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.376+0000 7f8530ff9640 10 monclient: dump: 2024-03-25T18:32:06.379 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:32:06.379 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:32:06.379 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:06.379 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:06.379 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:06.379 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:32:06.379 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:32:06.379 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:32:06.379 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:32:06.379 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:32:06.379 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.376+0000 7f85327fc640 4 set_mon_vals no callback set 2024-03-25T18:32:06.379 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.377+0000 7f8530ff9640 1 -- 172.21.15.155:0/833238670 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7f852801ab60 con 0x7f85340b4d60 2024-03-25T18:32:06.380 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.378+0000 7f8533fff640 10 monclient: get_auth_request con 0x7f852006ded0 auth_method 0 2024-03-25T18:32:06.380 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.378+0000 7f8530ff9640 10 client.4565.objecter ms_handle_connect 0x7f852006ded0 2024-03-25T18:32:06.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.388+0000 7f8530ff9640 1 -- 172.21.15.155:0/833238670 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7f8528054830 con 0x7f85340b4d60 2024-03-25T18:32:06.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.388+0000 7f8530ff9640 10 client.4565.objecter ms_dispatch 0x7f8534000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:32:06.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.388+0000 7f8530ff9640 3 client.4565.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:32:06.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.388+0000 7f8530ff9640 3 client.4565.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:32:06.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.388+0000 7f8530ff9640 20 client.4565.objecter dump_active .. 0 homeless 2024-03-25T18:32:06.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.388+0000 7f8530ff9640 1 -- 172.21.15.155:0/833238670 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8528050170 con 0x7f85340b4d60 2024-03-25T18:32:06.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.388+0000 7f8530ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:06.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.388+0000 7f8530ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:32:06.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.638+0000 7f853a54d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:06.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.638+0000 7f853a54d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:06.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.638+0000 7f853a54d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:32:06.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.638+0000 7f853a54d640 1 -- 172.21.15.155:0/833238670 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f85340627b0 con 0x7f85340b4d60 2024-03-25T18:32:06.681 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.679+0000 7f8530ff9640 1 -- 172.21.15.155:0/833238670 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7f8528052270 con 0x7f85340b4d60 2024-03-25T18:32:06.681 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.680+0000 7f8530ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:06.681 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.680+0000 7f8530ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:32:06.681 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:32:06.681 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:32:06.684 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.682+0000 7f853a54d640 10 librados: watch_flush enter 2024-03-25T18:32:06.684 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.682+0000 7f853a54d640 10 librados: watch_flush exit 2024-03-25T18:32:06.684 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.682+0000 7f853a54d640 20 client.4565.objecter shutdown clearing up homeless session... 2024-03-25T18:32:06.684 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.682+0000 7f853a54d640 10 client.4565.objecter successfully canceled tick 2024-03-25T18:32:06.684 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.682+0000 7f853a54d640 1 -- 172.21.15.155:0/833238670 >> v1:172.21.15.167:6800/2919070503 conn(0x7f852006ded0 legacy=0x7f8520070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:06.684 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.682+0000 7f853a54d640 10 monclient: shutdown 2024-03-25T18:32:06.684 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.682+0000 7f853a54d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:06.685 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.682+0000 7f853a54d640 1 -- 172.21.15.155:0/833238670 >> v1:172.21.15.155:6790/0 conn(0x7f85340b4d60 legacy=0x7f853412ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:06.685 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.682+0000 7f853a54d640 1 -- 172.21.15.155:0/833238670 shutdown_connections 2024-03-25T18:32:06.685 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.682+0000 7f853a54d640 1 -- 172.21.15.155:0/833238670 >> 172.21.15.155:0/833238670 conn(0x7f85340617a0 msgr2=0x7f85340b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:06.685 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.682+0000 7f853a54d640 1 -- 172.21.15.155:0/833238670 shutdown_connections 2024-03-25T18:32:06.685 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.682+0000 7f853a54d640 1 -- 172.21.15.155:0/833238670 wait complete. 2024-03-25T18:32:06.685 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:06.682+0000 7f853a54d640 1 librados: shutdown 2024-03-25T18:32:06.696 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:32:06.835 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:07:40.440 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:06.842 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:07:40.446 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:06.870 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:07:40.474 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:06.885 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:07:40.464 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:06.890 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:07:40.469 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:06.893 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:07:40.498 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:06.896 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:07:40.501 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:06.917 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:07:40.496 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:06.920 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:07:40.525 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:13.502 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:32:13.617 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.615+0000 7ffa1fd2b640 10 monclient: get_monmap_and_config 2024-03-25T18:32:13.617 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.615+0000 7ffa1fd2b640 10 monclient: build_initial_monmap 2024-03-25T18:32:13.617 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.615+0000 7ffa1fd2b640 10 monclient: monmap: 2024-03-25T18:32:13.617 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:32:13.618 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:32:13.618 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:32:13.616918+0000 2024-03-25T18:32:13.618 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:32:13.616918+0000 2024-03-25T18:32:13.618 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:13.618 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:32:13.618 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:32:13.618 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:32:13.618 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:32:13.618 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:32:13.619 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.616+0000 7ffa1fd2b640 1 Processor -- start 2024-03-25T18:32:13.619 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.616+0000 7ffa1fd2b640 1 -- start start 2024-03-25T18:32:13.619 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.616+0000 7ffa1fd2b640 10 monclient: init 2024-03-25T18:32:13.619 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.617+0000 7ffa1fd2b640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:13.619 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.617+0000 7ffa1fd2b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:32:13.619 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.617+0000 7ffa1fd2b640 10 monclient(hunting): picked mon.noname-a con 0x7ffa180a98f0 addr v1:172.21.15.155:6789/0 2024-03-25T18:32:13.619 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.617+0000 7ffa1fd2b640 10 monclient(hunting): picked mon.noname-b con 0x7ffa1813f710 addr v1:172.21.15.155:6790/0 2024-03-25T18:32:13.620 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.617+0000 7ffa1fd2b640 10 monclient(hunting): picked mon.noname-c con 0x7ffa181430c0 addr v1:172.21.15.167:6789/0 2024-03-25T18:32:13.620 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.617+0000 7ffa1fd2b640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa18142520 con 0x7ffa180a98f0 2024-03-25T18:32:13.620 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.617+0000 7ffa1fd2b640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa180b28a0 con 0x7ffa181430c0 2024-03-25T18:32:13.620 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.617+0000 7ffa1fd2b640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa180b2a30 con 0x7ffa1813f710 2024-03-25T18:32:13.620 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.617+0000 7ffa1fd2b640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:13.620 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.617+0000 7ffa1fd2b640 10 monclient(hunting): authenticate will time out at 7979.927612s 2024-03-25T18:32:13.623 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.622+0000 7ffa1d2c6640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7ffa1813f710 0x7ffa181429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:44234/0 (socket says 172.21.15.155:44234) 2024-03-25T18:32:13.623 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.622+0000 7ffa1d2c6640 1 -- 172.21.15.155:0/209932956 learned_addr learned my addr 172.21.15.155:0/209932956 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:32:13.634 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.633+0000 7ffa1cac5640 1 -- 172.21.15.155:0/209932956 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1084987442 0 0) 0x7ffa180b28a0 con 0x7ffa181430c0 2024-03-25T18:32:13.634 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.633+0000 7ffa1cac5640 10 monclient(hunting): _negotiate 2024-03-25T18:32:13.634 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.633+0000 7ffa1cac5640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:13.634 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.633+0000 7ffa1cac5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:13.634 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.633+0000 7ffa1cac5640 10 monclient(hunting): my global_id is 4555 2024-03-25T18:32:13.634 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.633+0000 7ffa1cac5640 1 -- 172.21.15.155:0/209932956 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa04003520 con 0x7ffa181430c0 2024-03-25T18:32:13.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.636+0000 7ffa1cac5640 1 -- 172.21.15.155:0/209932956 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1306299787 0 0) 0x7ffa18142520 con 0x7ffa180a98f0 2024-03-25T18:32:13.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.636+0000 7ffa1cac5640 10 monclient(hunting): _negotiate 2024-03-25T18:32:13.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.636+0000 7ffa1cac5640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:13.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.636+0000 7ffa1cac5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:13.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.636+0000 7ffa1cac5640 10 monclient(hunting): my global_id is 4560 2024-03-25T18:32:13.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.636+0000 7ffa1cac5640 1 -- 172.21.15.155:0/209932956 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa180b28a0 con 0x7ffa180a98f0 2024-03-25T18:32:13.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.637+0000 7ffa1cac5640 1 -- 172.21.15.155:0/209932956 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2885567663 0 0) 0x7ffa180b2a30 con 0x7ffa1813f710 2024-03-25T18:32:13.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.637+0000 7ffa1cac5640 10 monclient(hunting): _negotiate 2024-03-25T18:32:13.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.637+0000 7ffa1cac5640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:13.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.637+0000 7ffa1cac5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:13.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.637+0000 7ffa1cac5640 10 monclient(hunting): my global_id is 4568 2024-03-25T18:32:13.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.637+0000 7ffa1cac5640 1 -- 172.21.15.155:0/209932956 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa18142520 con 0x7ffa1813f710 2024-03-25T18:32:13.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.645+0000 7ffa1cac5640 1 -- 172.21.15.155:0/209932956 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3436411230 0 0) 0x7ffa180b28a0 con 0x7ffa180a98f0 2024-03-25T18:32:13.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.645+0000 7ffa1cac5640 1 -- 172.21.15.155:0/209932956 >> v1:172.21.15.155:6790/0 conn(0x7ffa1813f710 legacy=0x7ffa181429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:13.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.645+0000 7ffa1cac5640 1 -- 172.21.15.155:0/209932956 >> v1:172.21.15.167:6789/0 conn(0x7ffa181430c0 legacy=0x7ffa1814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:13.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.645+0000 7ffa1cac5640 10 monclient: _finish_hunting 0 2024-03-25T18:32:13.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.645+0000 7ffa1cac5640 1 monclient: found mon.noname-a 2024-03-25T18:32:13.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.645+0000 7ffa1cac5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:13.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.645+0000 7ffa1cac5640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-25T18:32:13.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.645+0000 7ffa1cac5640 1 -- 172.21.15.155:0/209932956 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa180b44c0 con 0x7ffa180a98f0 2024-03-25T18:32:13.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.645+0000 7ffa1cac5640 10 monclient: _finish_auth 0 2024-03-25T18:32:13.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.645+0000 7ffa1cac5640 10 monclient: _check_auth_tickets 2024-03-25T18:32:13.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.645+0000 7ffa1cac5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:13.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.645+0000 7ffa1cac5640 1 -- 172.21.15.155:0/209932956 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7ffa14001670 con 0x7ffa180a98f0 2024-03-25T18:32:13.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.645+0000 7ffa1cac5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:13.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.645+0000 7ffa1cac5640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:32:13.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.645+0000 7ffa1cac5640 10 monclient: dump: 2024-03-25T18:32:13.647 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:32:13.648 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:32:13.648 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:13.648 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:13.648 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:13.648 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:32:13.648 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:32:13.648 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:32:13.648 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:32:13.648 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:32:13.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.645+0000 7ffa1fd2b640 5 monclient: authenticate success, global_id 4560 2024-03-25T18:32:13.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.645+0000 7ffa1fd2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:13.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.658+0000 7ffa1cac5640 1 -- 172.21.15.155:0/209932956 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffa140017d0 con 0x7ffa180a98f0 2024-03-25T18:32:13.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.658+0000 7ffa1cac5640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:13.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.658+0000 7ffa1cac5640 1 -- 172.21.15.155:0/209932956 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7ffa14001be0 con 0x7ffa180a98f0 2024-03-25T18:32:13.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.658+0000 7ffa1cac5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:13.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.658+0000 7ffa1cac5640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:32:13.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.658+0000 7ffa1cac5640 10 monclient: dump: 2024-03-25T18:32:13.660 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:32:13.660 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:32:13.660 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:13.660 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:13.660 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:13.660 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:32:13.660 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:32:13.661 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:32:13.661 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:32:13.661 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:32:13.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.658+0000 7ffa1fd2b640 10 monclient: get_monmap_and_config success 2024-03-25T18:32:13.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.658+0000 7ffa1fd2b640 4 set_mon_vals no callback set 2024-03-25T18:32:13.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.659+0000 7ffa1fd2b640 10 monclient: shutdown 2024-03-25T18:32:13.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.659+0000 7ffa1fd2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:13.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.659+0000 7ffa1fd2b640 1 -- 172.21.15.155:0/209932956 >> v1:172.21.15.155:6789/0 conn(0x7ffa180a98f0 legacy=0x7ffa180a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:13.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.659+0000 7ffa1fd2b640 1 -- 172.21.15.155:0/209932956 shutdown_connections 2024-03-25T18:32:13.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.659+0000 7ffa1fd2b640 1 -- 172.21.15.155:0/209932956 >> 172.21.15.155:0/209932956 conn(0x7ffa180b7ea0 msgr2=0x7ffa180b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:13.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.660+0000 7ffa1fd2b640 1 -- 172.21.15.155:0/209932956 shutdown_connections 2024-03-25T18:32:13.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.660+0000 7ffa1fd2b640 1 -- 172.21.15.155:0/209932956 wait complete. 2024-03-25T18:32:13.663 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.661+0000 7ffa1fd2b640 10 monclient: build_initial_monmap 2024-03-25T18:32:13.663 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.661+0000 7ffa1fd2b640 10 monclient: monmap: 2024-03-25T18:32:13.663 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:32:13.663 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:32:13.663 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:32:13.663 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:32:13.663 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:13.663 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:32:13.663 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:32:13.663 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:32:13.664 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:32:13.664 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:32:13.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.661+0000 7ffa1fd2b640 1 librados: starting msgr at 2024-03-25T18:32:13.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.661+0000 7ffa1fd2b640 1 librados: starting objecter 2024-03-25T18:32:13.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.661+0000 7ffa1fd2b640 1 Processor -- start 2024-03-25T18:32:13.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.661+0000 7ffa1fd2b640 1 -- start start 2024-03-25T18:32:13.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.661+0000 7ffa1fd2b640 1 librados: setting wanted keys 2024-03-25T18:32:13.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.661+0000 7ffa1fd2b640 1 librados: calling monclient init 2024-03-25T18:32:13.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.661+0000 7ffa1fd2b640 10 monclient: init 2024-03-25T18:32:13.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.662+0000 7ffa1fd2b640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:13.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.662+0000 7ffa1fd2b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:32:13.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.662+0000 7ffa1fd2b640 10 monclient(hunting): picked mon.noname-a con 0x7ffa180a98f0 addr v1:172.21.15.155:6789/0 2024-03-25T18:32:13.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.662+0000 7ffa1fd2b640 10 monclient(hunting): picked mon.noname-c con 0x7ffa1813f710 addr v1:172.21.15.155:6790/0 2024-03-25T18:32:13.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.662+0000 7ffa1fd2b640 10 monclient(hunting): picked mon.noname-b con 0x7ffa181430c0 addr v1:172.21.15.167:6789/0 2024-03-25T18:32:13.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.662+0000 7ffa1fd2b640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa18139720 con 0x7ffa180a98f0 2024-03-25T18:32:13.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.662+0000 7ffa1fd2b640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa18138af0 con 0x7ffa181430c0 2024-03-25T18:32:13.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.663+0000 7ffa1fd2b640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa18138d60 con 0x7ffa1813f710 2024-03-25T18:32:13.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.663+0000 7ffa1fd2b640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:13.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.663+0000 7ffa1fd2b640 10 monclient(hunting): authenticate will time out at 7979.972790s 2024-03-25T18:32:13.670 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.669+0000 7ffa1d2c6640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7ffa1813f710 0x7ffa181372d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:44270/0 (socket says 172.21.15.155:44270) 2024-03-25T18:32:13.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.669+0000 7ffa1d2c6640 1 -- 172.21.15.155:0/4128303360 learned_addr learned my addr 172.21.15.155:0/4128303360 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:32:13.676 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.675+0000 7ffa0e7fc640 10 client.?.objecter ms_handle_connect 0x7ffa180a98f0 2024-03-25T18:32:13.676 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.675+0000 7ffa0e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:13.677 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.676+0000 7ffa0e7fc640 10 client.?.objecter ms_handle_connect 0x7ffa1813f710 2024-03-25T18:32:13.677 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.676+0000 7ffa0e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:13.686 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.685+0000 7ffa0e7fc640 10 client.?.objecter ms_handle_connect 0x7ffa181430c0 2024-03-25T18:32:13.686 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.685+0000 7ffa0e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:13.689 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.688+0000 7ffa0e7fc640 1 -- 172.21.15.155:0/4128303360 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1494929492 0 0) 0x7ffa18139720 con 0x7ffa180a98f0 2024-03-25T18:32:13.689 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.688+0000 7ffa0e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:32:13.689 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.688+0000 7ffa0e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:13.689 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.688+0000 7ffa0e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:13.689 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.688+0000 7ffa0e7fc640 10 monclient(hunting): my global_id is 4563 2024-03-25T18:32:13.689 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.688+0000 7ffa0e7fc640 1 -- 172.21.15.155:0/4128303360 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9fc004eb0 con 0x7ffa180a98f0 2024-03-25T18:32:13.690 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.688+0000 7ffa0e7fc640 1 -- 172.21.15.155:0/4128303360 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3782472166 0 0) 0x7ffa18138d60 con 0x7ffa1813f710 2024-03-25T18:32:13.690 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.688+0000 7ffa0e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:32:13.690 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.688+0000 7ffa0e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:13.690 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.688+0000 7ffa0e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:13.690 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.688+0000 7ffa0e7fc640 10 monclient(hunting): my global_id is 4571 2024-03-25T18:32:13.690 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.688+0000 7ffa0e7fc640 1 -- 172.21.15.155:0/4128303360 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa18139720 con 0x7ffa1813f710 2024-03-25T18:32:13.694 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.693+0000 7ffa0e7fc640 1 -- 172.21.15.155:0/4128303360 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1963600593 0 0) 0x7ffa18138af0 con 0x7ffa181430c0 2024-03-25T18:32:13.694 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.693+0000 7ffa0e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:32:13.694 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.693+0000 7ffa0e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:13.694 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.693+0000 7ffa0e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:13.695 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.693+0000 7ffa0e7fc640 10 monclient(hunting): my global_id is 4558 2024-03-25T18:32:13.695 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.693+0000 7ffa0e7fc640 1 -- 172.21.15.155:0/4128303360 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa18138d60 con 0x7ffa181430c0 2024-03-25T18:32:13.705 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.703+0000 7ffa0e7fc640 1 -- 172.21.15.155:0/4128303360 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1717448044 0 0) 0x7ffa18138d60 con 0x7ffa181430c0 2024-03-25T18:32:13.705 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.703+0000 7ffa0e7fc640 1 -- 172.21.15.155:0/4128303360 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa18138af0 con 0x7ffa181430c0 2024-03-25T18:32:13.705 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.703+0000 7ffa0e7fc640 1 -- 172.21.15.155:0/4128303360 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7ffa10001a30 con 0x7ffa181430c0 2024-03-25T18:32:13.705 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.703+0000 7ffa0e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:13.705 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.703+0000 7ffa0e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:32:13.705 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.703+0000 7ffa0e7fc640 10 monclient(hunting): dump: 2024-03-25T18:32:13.705 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:32:13.705 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:32:13.705 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:13.705 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:13.705 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:13.705 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:32:13.705 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:32:13.705 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:32:13.705 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:32:13.706 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:32:13.706 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.703+0000 7ffa0e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:32:13.712 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.710+0000 7ffa0e7fc640 1 -- 172.21.15.155:0/4128303360 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2972398250 0 0) 0x7ffa18139720 con 0x7ffa1813f710 2024-03-25T18:32:13.712 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.710+0000 7ffa0e7fc640 1 -- 172.21.15.155:0/4128303360 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa18138d60 con 0x7ffa1813f710 2024-03-25T18:32:13.712 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.710+0000 7ffa0e7fc640 1 -- 172.21.15.155:0/4128303360 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7ffa080018f0 con 0x7ffa1813f710 2024-03-25T18:32:13.712 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.710+0000 7ffa0e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:13.712 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.710+0000 7ffa0e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:32:13.712 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.710+0000 7ffa0e7fc640 10 monclient(hunting): dump: 2024-03-25T18:32:13.712 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:32:13.712 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:32:13.712 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:13.712 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:13.712 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:13.712 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:32:13.712 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:32:13.712 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:32:13.713 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:32:13.713 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:32:13.717 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.716+0000 7ffa0e7fc640 1 -- 172.21.15.155:0/4128303360 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1079845882 0 0) 0x7ff9fc004eb0 con 0x7ffa180a98f0 2024-03-25T18:32:13.717 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.716+0000 7ffa0e7fc640 1 -- 172.21.15.155:0/4128303360 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa18139720 con 0x7ffa180a98f0 2024-03-25T18:32:13.717 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.716+0000 7ffa0e7fc640 1 -- 172.21.15.155:0/4128303360 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7ffa14001ba0 con 0x7ffa180a98f0 2024-03-25T18:32:13.717 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.716+0000 7ffa0e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:13.717 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.716+0000 7ffa0e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:32:13.717 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.716+0000 7ffa0e7fc640 10 monclient(hunting): dump: 2024-03-25T18:32:13.718 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:32:13.718 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:32:13.718 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:13.718 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:13.718 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:13.718 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:32:13.718 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:32:13.718 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:32:13.718 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:32:13.718 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:32:13.722 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.720+0000 7ffa0e7fc640 1 -- 172.21.15.155:0/4128303360 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1092941892 0 0) 0x7ffa18138d60 con 0x7ffa1813f710 2024-03-25T18:32:13.722 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.720+0000 7ffa0e7fc640 1 -- 172.21.15.155:0/4128303360 >> v1:172.21.15.167:6789/0 conn(0x7ffa181430c0 legacy=0x7ffa180b0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:13.722 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.720+0000 7ffa0e7fc640 1 -- 172.21.15.155:0/4128303360 >> v1:172.21.15.155:6789/0 conn(0x7ffa180a98f0 legacy=0x7ffa18136690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:13.722 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.720+0000 7ffa0e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:32:13.722 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.720+0000 7ffa0e7fc640 1 monclient: found mon.c 2024-03-25T18:32:13.722 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.720+0000 7ffa0e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:13.722 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.720+0000 7ffa0e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:32:13.722 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.720+0000 7ffa0e7fc640 1 -- 172.21.15.155:0/4128303360 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa181379e0 con 0x7ffa1813f710 2024-03-25T18:32:13.722 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.720+0000 7ffa0e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:32:13.722 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.720+0000 7ffa0e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:32:13.722 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.720+0000 7ffa0e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:13.722 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.720+0000 7ffa1fd2b640 5 monclient: authenticate success, global_id 4571 2024-03-25T18:32:13.723 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.720+0000 7ffa1fd2b640 10 monclient: _renew_subs 2024-03-25T18:32:13.723 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.720+0000 7ffa1fd2b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:32:13.723 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.720+0000 7ffa1fd2b640 1 -- 172.21.15.155:0/4128303360 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa181da8f0 con 0x7ffa1813f710 2024-03-25T18:32:13.723 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.720+0000 7ffa1fd2b640 10 client.4571.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:32:13.723 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.720+0000 7ffa1fd2b640 10 monclient: _renew_subs 2024-03-25T18:32:13.723 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.720+0000 7ffa1fd2b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:32:13.723 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.720+0000 7ffa1fd2b640 1 -- 172.21.15.155:0/4128303360 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa1813c380 con 0x7ffa1813f710 2024-03-25T18:32:13.723 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.720+0000 7ffa1fd2b640 1 librados: init done 2024-03-25T18:32:13.723 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.721+0000 7ffa1fd2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:13.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.721+0000 7ffa1fd2b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:13.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.721+0000 7ffa1fd2b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:32:13.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:13.721+0000 7ffa1fd2b640 1 -- 172.21.15.155:0/4128303360 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa18074cc0 con 0x7ffa1813f710 2024-03-25T18:32:14.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:14.032+0000 7ffa0e7fc640 1 -- 172.21.15.155:0/4128303360 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffa080047e0 con 0x7ffa1813f710 2024-03-25T18:32:14.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:14.032+0000 7ffa0e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:14.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:14.032+0000 7ffa0e7fc640 1 -- 172.21.15.155:0/4128303360 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7ffa08004b70 con 0x7ffa1813f710 2024-03-25T18:32:14.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:14.032+0000 7ffa0e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:14.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:14.032+0000 7ffa0e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:32:14.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:14.032+0000 7ffa0e7fc640 10 monclient: dump: 2024-03-25T18:32:14.035 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:32:14.035 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:32:14.035 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:14.035 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:14.035 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:14.035 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:32:14.035 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:32:14.035 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:32:14.035 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:32:14.035 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:32:14.035 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:14.032+0000 7ffa0ffff640 4 set_mon_vals no callback set 2024-03-25T18:32:14.036 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:14.033+0000 7ffa0e7fc640 1 -- 172.21.15.155:0/4128303360 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7ffa08004d90 con 0x7ffa1813f710 2024-03-25T18:32:14.036 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:14.034+0000 7ffa1dac7640 10 monclient: get_auth_request con 0x7ff9fc06e0e0 auth_method 0 2024-03-25T18:32:14.036 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:14.035+0000 7ffa0e7fc640 10 client.4571.objecter ms_handle_connect 0x7ff9fc06e0e0 2024-03-25T18:32:14.042 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:14.040+0000 7ffa0e7fc640 1 -- 172.21.15.155:0/4128303360 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7ffa08054770 con 0x7ffa1813f710 2024-03-25T18:32:14.042 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:14.040+0000 7ffa0e7fc640 10 client.4571.objecter ms_dispatch 0x7ffa18000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:32:14.042 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:14.040+0000 7ffa0e7fc640 3 client.4571.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:32:14.042 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:14.040+0000 7ffa0e7fc640 3 client.4571.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:32:14.042 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:14.041+0000 7ffa0e7fc640 20 client.4571.objecter dump_active .. 0 homeless 2024-03-25T18:32:14.042 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:14.041+0000 7ffa0e7fc640 1 -- 172.21.15.155:0/4128303360 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffa08050280 con 0x7ffa1813f710 2024-03-25T18:32:14.042 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:14.041+0000 7ffa0e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:14.042 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:14.041+0000 7ffa0e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:32:14.268 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:14.266+0000 7ffa1fd2b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:14.268 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:14.266+0000 7ffa1fd2b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:14.268 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:14.266+0000 7ffa1fd2b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:32:14.268 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:14.266+0000 7ffa1fd2b640 1 -- 172.21.15.155:0/4128303360 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffa18136df0 con 0x7ffa1813f710 2024-03-25T18:32:14.304 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:14.302+0000 7ffa0e7fc640 1 -- 172.21.15.155:0/4128303360 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7ffa080521b0 con 0x7ffa1813f710 2024-03-25T18:32:14.304 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:14.302+0000 7ffa0e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:14.304 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:14.302+0000 7ffa0e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:32:14.304 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:32:14.304 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:32:14.306 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:14.305+0000 7ffa1fd2b640 10 librados: watch_flush enter 2024-03-25T18:32:14.306 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:14.305+0000 7ffa1fd2b640 10 librados: watch_flush exit 2024-03-25T18:32:14.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:14.305+0000 7ffa1fd2b640 20 client.4571.objecter shutdown clearing up homeless session... 2024-03-25T18:32:14.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:14.305+0000 7ffa1fd2b640 10 client.4571.objecter successfully canceled tick 2024-03-25T18:32:14.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:14.305+0000 7ffa1fd2b640 1 -- 172.21.15.155:0/4128303360 >> v1:172.21.15.167:6800/2919070503 conn(0x7ff9fc06e0e0 legacy=0x7ff9fc0705a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:14.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:14.305+0000 7ffa1fd2b640 10 monclient: shutdown 2024-03-25T18:32:14.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:14.305+0000 7ffa1fd2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:14.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:14.305+0000 7ffa1fd2b640 1 -- 172.21.15.155:0/4128303360 >> v1:172.21.15.155:6790/0 conn(0x7ffa1813f710 legacy=0x7ffa181372d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:14.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:14.305+0000 7ffa1fd2b640 1 -- 172.21.15.155:0/4128303360 shutdown_connections 2024-03-25T18:32:14.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:14.305+0000 7ffa1fd2b640 1 -- 172.21.15.155:0/4128303360 >> 172.21.15.155:0/4128303360 conn(0x7ffa180b7ea0 msgr2=0x7ffa18074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:14.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:14.305+0000 7ffa1fd2b640 1 -- 172.21.15.155:0/4128303360 shutdown_connections 2024-03-25T18:32:14.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:14.305+0000 7ffa1fd2b640 1 -- 172.21.15.155:0/4128303360 wait complete. 2024-03-25T18:32:14.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:14.305+0000 7ffa1fd2b640 1 librados: shutdown 2024-03-25T18:32:14.318 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:32:21.125 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:32:21.239 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.238+0000 7fd378ce0640 10 monclient: get_monmap_and_config 2024-03-25T18:32:21.240 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.238+0000 7fd378ce0640 10 monclient: build_initial_monmap 2024-03-25T18:32:21.240 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.238+0000 7fd378ce0640 10 monclient: monmap: 2024-03-25T18:32:21.240 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:32:21.240 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:32:21.240 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:32:21.239563+0000 2024-03-25T18:32:21.240 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:32:21.239563+0000 2024-03-25T18:32:21.240 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:21.240 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:32:21.240 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:32:21.240 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:32:21.240 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:32:21.240 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:32:21.242 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.239+0000 7fd378ce0640 1 Processor -- start 2024-03-25T18:32:21.242 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.239+0000 7fd378ce0640 1 -- start start 2024-03-25T18:32:21.242 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.239+0000 7fd378ce0640 10 monclient: init 2024-03-25T18:32:21.242 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.240+0000 7fd378ce0640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:21.242 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.240+0000 7fd378ce0640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:32:21.242 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.240+0000 7fd378ce0640 10 monclient(hunting): picked mon.noname-c con 0x7fd3740a96e0 addr v1:172.21.15.167:6789/0 2024-03-25T18:32:21.242 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.240+0000 7fd378ce0640 10 monclient(hunting): picked mon.noname-a con 0x7fd3740b1b10 addr v1:172.21.15.155:6789/0 2024-03-25T18:32:21.242 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.240+0000 7fd378ce0640 10 monclient(hunting): picked mon.noname-b con 0x7fd374146060 addr v1:172.21.15.155:6790/0 2024-03-25T18:32:21.242 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.240+0000 7fd378ce0640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd374140040 con 0x7fd3740b1b10 2024-03-25T18:32:21.242 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.240+0000 7fd378ce0640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3740b1fa0 con 0x7fd3740a96e0 2024-03-25T18:32:21.243 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.240+0000 7fd378ce0640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3740b24c0 con 0x7fd374146060 2024-03-25T18:32:21.243 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.240+0000 7fd378ce0640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:21.243 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.240+0000 7fd378ce0640 10 monclient(hunting): authenticate will time out at 7987.549863s 2024-03-25T18:32:21.243 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.240+0000 7fd37259c640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7fd3740a96e0 0x7fd3740b0ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:56498/0 (socket says 172.21.15.155:56498) 2024-03-25T18:32:21.243 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.240+0000 7fd37259c640 1 -- 172.21.15.155:0/3004399600 learned_addr learned my addr 172.21.15.155:0/3004399600 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:32:21.309 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.308+0000 7fd371d9b640 1 -- 172.21.15.155:0/3004399600 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4230059022 0 0) 0x7fd374140040 con 0x7fd3740b1b10 2024-03-25T18:32:21.309 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.308+0000 7fd371d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:32:21.310 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.308+0000 7fd371d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:21.310 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.308+0000 7fd371d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:21.310 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.308+0000 7fd371d9b640 10 monclient(hunting): my global_id is 4566 2024-03-25T18:32:21.310 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.308+0000 7fd371d9b640 1 -- 172.21.15.155:0/3004399600 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd360003520 con 0x7fd3740b1b10 2024-03-25T18:32:21.321 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.320+0000 7fd371d9b640 1 -- 172.21.15.155:0/3004399600 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2880396862 0 0) 0x7fd360003520 con 0x7fd3740b1b10 2024-03-25T18:32:21.322 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.320+0000 7fd371d9b640 1 -- 172.21.15.155:0/3004399600 >> v1:172.21.15.155:6790/0 conn(0x7fd374146060 legacy=0x7fd37414dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:21.322 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.320+0000 7fd371d9b640 1 -- 172.21.15.155:0/3004399600 >> v1:172.21.15.167:6789/0 conn(0x7fd3740a96e0 legacy=0x7fd3740b0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:21.322 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.320+0000 7fd371d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:32:21.322 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.320+0000 7fd371d9b640 1 monclient: found mon.noname-a 2024-03-25T18:32:21.322 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.320+0000 7fd371d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:21.322 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.320+0000 7fd371d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-25T18:32:21.322 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.320+0000 7fd371d9b640 1 -- 172.21.15.155:0/3004399600 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3740b15b0 con 0x7fd3740b1b10 2024-03-25T18:32:21.322 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.320+0000 7fd371d9b640 10 monclient: _finish_auth 0 2024-03-25T18:32:21.322 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.320+0000 7fd371d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:32:21.322 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.320+0000 7fd371d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:21.322 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.320+0000 7fd371d9b640 1 -- 172.21.15.155:0/3004399600 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fd35c001700 con 0x7fd3740b1b10 2024-03-25T18:32:21.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.320+0000 7fd371d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:21.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.320+0000 7fd371d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:32:21.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.320+0000 7fd371d9b640 10 monclient: dump: 2024-03-25T18:32:21.323 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:32:21.323 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:32:21.323 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:21.323 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:21.323 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:21.323 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:32:21.323 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:32:21.323 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:32:21.323 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:32:21.323 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:32:21.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.320+0000 7fd378ce0640 5 monclient: authenticate success, global_id 4566 2024-03-25T18:32:21.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.320+0000 7fd378ce0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:21.327 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.326+0000 7fd371d9b640 1 -- 172.21.15.155:0/3004399600 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd35c001860 con 0x7fd3740b1b10 2024-03-25T18:32:21.327 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.326+0000 7fd371d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:21.327 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.326+0000 7fd378ce0640 10 monclient: get_monmap_and_config success 2024-03-25T18:32:21.327 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.326+0000 7fd378ce0640 4 set_mon_vals no callback set 2024-03-25T18:32:21.328 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.326+0000 7fd371d9b640 1 -- 172.21.15.155:0/3004399600 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fd35c001c70 con 0x7fd3740b1b10 2024-03-25T18:32:21.329 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.326+0000 7fd371d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:21.329 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.326+0000 7fd371d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:32:21.329 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.326+0000 7fd371d9b640 10 monclient: dump: 2024-03-25T18:32:21.329 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:32:21.329 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:32:21.329 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:21.329 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:21.329 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:21.329 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:32:21.329 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:32:21.329 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:32:21.329 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:32:21.329 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:32:21.329 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.327+0000 7fd378ce0640 10 monclient: shutdown 2024-03-25T18:32:21.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.327+0000 7fd378ce0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:21.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.327+0000 7fd378ce0640 1 -- 172.21.15.155:0/3004399600 >> v1:172.21.15.155:6789/0 conn(0x7fd3740b1b10 legacy=0x7fd374145950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:21.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.327+0000 7fd378ce0640 1 -- 172.21.15.155:0/3004399600 shutdown_connections 2024-03-25T18:32:21.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.327+0000 7fd378ce0640 1 -- 172.21.15.155:0/3004399600 >> 172.21.15.155:0/3004399600 conn(0x7fd3740b7ea0 msgr2=0x7fd3740b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:21.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.327+0000 7fd378ce0640 1 -- 172.21.15.155:0/3004399600 shutdown_connections 2024-03-25T18:32:21.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.327+0000 7fd378ce0640 1 -- 172.21.15.155:0/3004399600 wait complete. 2024-03-25T18:32:21.332 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.328+0000 7fd378ce0640 10 monclient: build_initial_monmap 2024-03-25T18:32:21.332 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.328+0000 7fd378ce0640 10 monclient: monmap: 2024-03-25T18:32:21.332 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:32:21.332 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:32:21.332 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:32:21.332 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:32:21.332 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:21.332 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:32:21.332 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:32:21.332 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:32:21.332 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:32:21.332 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:32:21.333 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.328+0000 7fd378ce0640 1 librados: starting msgr at 2024-03-25T18:32:21.333 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.328+0000 7fd378ce0640 1 librados: starting objecter 2024-03-25T18:32:21.333 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.328+0000 7fd378ce0640 1 Processor -- start 2024-03-25T18:32:21.333 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.328+0000 7fd378ce0640 1 -- start start 2024-03-25T18:32:21.333 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.328+0000 7fd378ce0640 1 librados: setting wanted keys 2024-03-25T18:32:21.333 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.328+0000 7fd378ce0640 1 librados: calling monclient init 2024-03-25T18:32:21.333 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.328+0000 7fd378ce0640 10 monclient: init 2024-03-25T18:32:21.333 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.329+0000 7fd378ce0640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:21.333 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.329+0000 7fd378ce0640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:32:21.333 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.329+0000 7fd378ce0640 10 monclient(hunting): picked mon.noname-a con 0x7fd3740a96e0 addr v1:172.21.15.155:6789/0 2024-03-25T18:32:21.333 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.329+0000 7fd378ce0640 10 monclient(hunting): picked mon.noname-c con 0x7fd374146060 addr v1:172.21.15.155:6790/0 2024-03-25T18:32:21.333 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.329+0000 7fd378ce0640 10 monclient(hunting): picked mon.noname-b con 0x7fd3741377c0 addr v1:172.21.15.167:6789/0 2024-03-25T18:32:21.333 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.329+0000 7fd378ce0640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd374141b20 con 0x7fd3740a96e0 2024-03-25T18:32:21.333 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.329+0000 7fd378ce0640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd374137bd0 con 0x7fd3741377c0 2024-03-25T18:32:21.333 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.329+0000 7fd378ce0640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd37413ae90 con 0x7fd374146060 2024-03-25T18:32:21.333 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.329+0000 7fd378ce0640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:21.334 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.329+0000 7fd378ce0640 10 monclient(hunting): authenticate will time out at 7987.639261s 2024-03-25T18:32:21.343 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.342+0000 7fd37259c640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fd3740a96e0 0x7fd3741370b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:59490/0 (socket says 172.21.15.155:59490) 2024-03-25T18:32:21.343 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.342+0000 7fd37259c640 1 -- 172.21.15.155:0/3111640257 learned_addr learned my addr 172.21.15.155:0/3111640257 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:32:21.343 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.342+0000 7fd36affd640 10 client.?.objecter ms_handle_connect 0x7fd3740a96e0 2024-03-25T18:32:21.343 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.342+0000 7fd36affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:21.347 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.346+0000 7fd36affd640 10 client.?.objecter ms_handle_connect 0x7fd3741377c0 2024-03-25T18:32:21.347 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.346+0000 7fd36affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:21.355 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.354+0000 7fd36affd640 10 client.?.objecter ms_handle_connect 0x7fd374146060 2024-03-25T18:32:21.355 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.354+0000 7fd36affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:21.363 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.362+0000 7fd36affd640 1 -- 172.21.15.155:0/3111640257 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2993491807 0 0) 0x7fd374141b20 con 0x7fd3740a96e0 2024-03-25T18:32:21.363 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.362+0000 7fd36affd640 10 monclient(hunting): _negotiate 2024-03-25T18:32:21.363 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.362+0000 7fd36affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:21.363 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.362+0000 7fd36affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:21.363 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.362+0000 7fd36affd640 10 monclient(hunting): my global_id is 4569 2024-03-25T18:32:21.363 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.362+0000 7fd36affd640 1 -- 172.21.15.155:0/3111640257 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd354004eb0 con 0x7fd3740a96e0 2024-03-25T18:32:21.420 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.419+0000 7fd36affd640 1 -- 172.21.15.155:0/3111640257 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3677669420 0 0) 0x7fd374137bd0 con 0x7fd3741377c0 2024-03-25T18:32:21.420 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.419+0000 7fd36affd640 10 monclient(hunting): _negotiate 2024-03-25T18:32:21.420 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.419+0000 7fd36affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:21.421 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.419+0000 7fd36affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:21.421 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.419+0000 7fd36affd640 10 monclient(hunting): my global_id is 4564 2024-03-25T18:32:21.421 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.419+0000 7fd36affd640 1 -- 172.21.15.155:0/3111640257 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd374141b20 con 0x7fd3741377c0 2024-03-25T18:32:21.423 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.422+0000 7fd36affd640 1 -- 172.21.15.155:0/3111640257 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4081473282 0 0) 0x7fd354004eb0 con 0x7fd3740a96e0 2024-03-25T18:32:21.423 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.422+0000 7fd36affd640 1 -- 172.21.15.155:0/3111640257 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd374137bd0 con 0x7fd3740a96e0 2024-03-25T18:32:21.424 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.422+0000 7fd36affd640 1 -- 172.21.15.155:0/3111640257 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fd3640018b0 con 0x7fd3740a96e0 2024-03-25T18:32:21.424 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.422+0000 7fd36affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:21.424 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.422+0000 7fd36affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:32:21.424 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.422+0000 7fd36affd640 10 monclient(hunting): dump: 2024-03-25T18:32:21.424 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:32:21.424 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:32:21.424 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:21.424 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:21.424 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:21.424 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:32:21.424 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:32:21.424 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:32:21.424 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:32:21.424 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:32:21.425 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.422+0000 7fd36affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:32:21.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.430+0000 7fd36affd640 1 -- 172.21.15.155:0/3111640257 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1465002916 0 0) 0x7fd374141b20 con 0x7fd3741377c0 2024-03-25T18:32:21.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.430+0000 7fd36affd640 1 -- 172.21.15.155:0/3111640257 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd354004eb0 con 0x7fd3741377c0 2024-03-25T18:32:21.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.430+0000 7fd36affd640 1 -- 172.21.15.155:0/3111640257 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fd36c001b00 con 0x7fd3741377c0 2024-03-25T18:32:21.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.430+0000 7fd36affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:21.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.430+0000 7fd36affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:32:21.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.430+0000 7fd36affd640 10 monclient(hunting): dump: 2024-03-25T18:32:21.432 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:32:21.432 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:32:21.432 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:21.432 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:21.432 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:21.433 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:32:21.433 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:32:21.433 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:32:21.433 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:32:21.433 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:32:21.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.433+0000 7fd36affd640 1 -- 172.21.15.155:0/3111640257 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2061170216 0 0) 0x7fd37413ae90 con 0x7fd374146060 2024-03-25T18:32:21.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.434+0000 7fd36affd640 10 monclient(hunting): _negotiate 2024-03-25T18:32:21.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.434+0000 7fd36affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:21.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.434+0000 7fd36affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:21.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.434+0000 7fd36affd640 10 monclient(hunting): my global_id is 4577 2024-03-25T18:32:21.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.434+0000 7fd36affd640 1 -- 172.21.15.155:0/3111640257 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd374141b20 con 0x7fd374146060 2024-03-25T18:32:21.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.454+0000 7fd36affd640 1 -- 172.21.15.155:0/3111640257 <== mon.1 v1:172.21.15.167:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 676299587 0 0) 0x7fd354004eb0 con 0x7fd3741377c0 2024-03-25T18:32:21.456 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.454+0000 7fd36affd640 1 -- 172.21.15.155:0/3111640257 >> v1:172.21.15.155:6790/0 conn(0x7fd374146060 legacy=0x7fd374139380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:21.456 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.454+0000 7fd36affd640 1 -- 172.21.15.155:0/3111640257 >> v1:172.21.15.155:6789/0 conn(0x7fd3740a96e0 legacy=0x7fd3741370b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:21.456 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.454+0000 7fd36affd640 10 monclient: _finish_hunting 0 2024-03-25T18:32:21.456 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.454+0000 7fd36affd640 1 monclient: found mon.b 2024-03-25T18:32:21.456 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.454+0000 7fd36affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:21.456 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.454+0000 7fd36affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:32:21.456 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.454+0000 7fd36affd640 1 -- 172.21.15.155:0/3111640257 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd374139a90 con 0x7fd3741377c0 2024-03-25T18:32:21.456 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.454+0000 7fd36affd640 10 monclient: _finish_auth 0 2024-03-25T18:32:21.456 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.454+0000 7fd36affd640 10 monclient: _check_auth_tickets 2024-03-25T18:32:21.456 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.454+0000 7fd36affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:21.457 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.454+0000 7fd378ce0640 5 monclient: authenticate success, global_id 4564 2024-03-25T18:32:21.457 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.454+0000 7fd378ce0640 10 monclient: _renew_subs 2024-03-25T18:32:21.457 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.454+0000 7fd378ce0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:32:21.457 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.454+0000 7fd378ce0640 1 -- 172.21.15.155:0/3111640257 --> v1:172.21.15.167:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd374139d60 con 0x7fd3741377c0 2024-03-25T18:32:21.457 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.454+0000 7fd378ce0640 10 client.4564.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:32:21.457 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.454+0000 7fd378ce0640 10 monclient: _renew_subs 2024-03-25T18:32:21.457 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.454+0000 7fd378ce0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:32:21.457 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.454+0000 7fd378ce0640 1 -- 172.21.15.155:0/3111640257 --> v1:172.21.15.167:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3741428a0 con 0x7fd3741377c0 2024-03-25T18:32:21.457 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.454+0000 7fd378ce0640 1 librados: init done 2024-03-25T18:32:21.458 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.455+0000 7fd378ce0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:21.458 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.455+0000 7fd378ce0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:21.458 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.455+0000 7fd378ce0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:32:21.458 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.455+0000 7fd378ce0640 1 -- 172.21.15.155:0/3111640257 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3740a9ac0 con 0x7fd3741377c0 2024-03-25T18:32:21.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.735+0000 7fd36affd640 1 -- 172.21.15.155:0/3111640257 <== mon.1 v1:172.21.15.167:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd36c001fb0 con 0x7fd3741377c0 2024-03-25T18:32:21.737 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.735+0000 7fd36affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:21.737 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.735+0000 7fd36affd640 1 -- 172.21.15.155:0/3111640257 <== mon.1 v1:172.21.15.167:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fd36c003b10 con 0x7fd3741377c0 2024-03-25T18:32:21.737 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.735+0000 7fd36affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:21.737 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.735+0000 7fd36affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:32:21.737 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.735+0000 7fd36affd640 10 monclient: dump: 2024-03-25T18:32:21.737 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:32:21.737 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:32:21.737 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:21.737 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:21.737 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:21.737 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:32:21.737 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:32:21.737 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:32:21.738 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:32:21.738 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:32:21.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.735+0000 7fd37159a640 4 set_mon_vals no callback set 2024-03-25T18:32:21.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.736+0000 7fd36affd640 1 -- 172.21.15.155:0/3111640257 <== mon.1 v1:172.21.15.167:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7fd36c003d30 con 0x7fd3741377c0 2024-03-25T18:32:21.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.736+0000 7fd37259c640 10 monclient: get_auth_request con 0x7fd35406df50 auth_method 0 2024-03-25T18:32:21.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.737+0000 7fd36affd640 10 client.4564.objecter ms_handle_connect 0x7fd35406df50 2024-03-25T18:32:21.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.741+0000 7fd36affd640 1 -- 172.21.15.155:0/3111640257 <== mon.1 v1:172.21.15.167:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7fd36c053750 con 0x7fd3741377c0 2024-03-25T18:32:21.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.741+0000 7fd36affd640 10 client.4564.objecter ms_dispatch 0x7fd374000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:32:21.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.741+0000 7fd36affd640 3 client.4564.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:32:21.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.741+0000 7fd36affd640 3 client.4564.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:32:21.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.741+0000 7fd36affd640 20 client.4564.objecter dump_active .. 0 homeless 2024-03-25T18:32:21.743 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.741+0000 7fd36affd640 1 -- 172.21.15.155:0/3111640257 <== mon.1 v1:172.21.15.167:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd36c04f260 con 0x7fd3741377c0 2024-03-25T18:32:21.743 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.741+0000 7fd36affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:21.743 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.741+0000 7fd36affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:32:21.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.992+0000 7fd378ce0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:21.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.992+0000 7fd378ce0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:21.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.992+0000 7fd378ce0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:32:21.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:21.992+0000 7fd378ce0640 1 -- 172.21.15.155:0/3111640257 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd3740627b0 con 0x7fd3741377c0 2024-03-25T18:32:22.017 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:22.016+0000 7fd36affd640 1 -- 172.21.15.155:0/3111640257 <== mon.1 v1:172.21.15.167:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7fd36c051190 con 0x7fd3741377c0 2024-03-25T18:32:22.017 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:22.016+0000 7fd36affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:22.017 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:22.016+0000 7fd36affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:32:22.017 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:32:22.017 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:32:22.020 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:22.018+0000 7fd378ce0640 10 librados: watch_flush enter 2024-03-25T18:32:22.020 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:22.018+0000 7fd378ce0640 10 librados: watch_flush exit 2024-03-25T18:32:22.020 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:22.018+0000 7fd378ce0640 20 client.4564.objecter shutdown clearing up homeless session... 2024-03-25T18:32:22.020 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:22.018+0000 7fd378ce0640 10 client.4564.objecter successfully canceled tick 2024-03-25T18:32:22.020 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:22.018+0000 7fd378ce0640 1 -- 172.21.15.155:0/3111640257 >> v1:172.21.15.167:6800/2919070503 conn(0x7fd35406df50 legacy=0x7fd354070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:22.020 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:22.018+0000 7fd378ce0640 10 monclient: shutdown 2024-03-25T18:32:22.020 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:22.018+0000 7fd378ce0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:22.020 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:22.018+0000 7fd378ce0640 1 -- 172.21.15.155:0/3111640257 >> v1:172.21.15.167:6789/0 conn(0x7fd3741377c0 legacy=0x7fd374141fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:22.020 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:22.018+0000 7fd378ce0640 1 -- 172.21.15.155:0/3111640257 shutdown_connections 2024-03-25T18:32:22.020 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:22.018+0000 7fd378ce0640 1 -- 172.21.15.155:0/3111640257 >> 172.21.15.155:0/3111640257 conn(0x7fd3740b7ea0 msgr2=0x7fd37413ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:22.020 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:22.018+0000 7fd378ce0640 1 -- 172.21.15.155:0/3111640257 shutdown_connections 2024-03-25T18:32:22.020 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:22.018+0000 7fd378ce0640 1 -- 172.21.15.155:0/3111640257 wait complete. 2024-03-25T18:32:22.020 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:22.018+0000 7fd378ce0640 1 librados: shutdown 2024-03-25T18:32:22.032 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:32:26.856 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:08:00.460 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:26.860 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:08:00.465 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:26.902 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:08:00.504 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:26.912 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:08:00.517 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:26.915 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:08:00.494 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:26.921 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:08:00.499 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:26.925 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:08:00.529 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:26.952 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:08:00.530 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:26.953 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:08:00.558 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:28.839 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:32:28.954 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.952+0000 7f9a4468d640 10 monclient: get_monmap_and_config 2024-03-25T18:32:28.954 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.953+0000 7f9a4468d640 10 monclient: build_initial_monmap 2024-03-25T18:32:28.954 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.953+0000 7f9a4468d640 10 monclient: monmap: 2024-03-25T18:32:28.954 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:32:28.954 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:32:28.955 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:32:28.953972+0000 2024-03-25T18:32:28.955 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:32:28.953972+0000 2024-03-25T18:32:28.955 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:28.955 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:32:28.955 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:32:28.955 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:32:28.955 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:32:28.955 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:32:28.956 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.953+0000 7f9a4468d640 1 Processor -- start 2024-03-25T18:32:28.956 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.953+0000 7f9a4468d640 1 -- start start 2024-03-25T18:32:28.956 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.953+0000 7f9a4468d640 10 monclient: init 2024-03-25T18:32:28.956 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.954+0000 7f9a4468d640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:28.956 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.954+0000 7f9a4468d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:32:28.957 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.954+0000 7f9a4468d640 10 monclient(hunting): picked mon.noname-b con 0x7f9a3c0a98f0 addr v1:172.21.15.155:6790/0 2024-03-25T18:32:28.957 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.954+0000 7f9a4468d640 10 monclient(hunting): picked mon.noname-a con 0x7f9a3c13f710 addr v1:172.21.15.155:6789/0 2024-03-25T18:32:28.957 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.954+0000 7f9a4468d640 10 monclient(hunting): picked mon.noname-c con 0x7f9a3c1430c0 addr v1:172.21.15.167:6789/0 2024-03-25T18:32:28.957 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.954+0000 7f9a4468d640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a3c142520 con 0x7f9a3c13f710 2024-03-25T18:32:28.957 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.954+0000 7f9a4468d640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a3c0b28a0 con 0x7f9a3c1430c0 2024-03-25T18:32:28.957 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.954+0000 7f9a4468d640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a3c0b2a30 con 0x7f9a3c0a98f0 2024-03-25T18:32:28.957 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.954+0000 7f9a4468d640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:28.957 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.954+0000 7f9a4468d640 10 monclient(hunting): authenticate will time out at 7995.264418s 2024-03-25T18:32:28.968 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.967+0000 7f9a42c2a640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f9a3c1430c0 0x7f9a3c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:39580/0 (socket says 172.21.15.155:39580) 2024-03-25T18:32:28.968 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.967+0000 7f9a42c2a640 1 -- 172.21.15.155:0/4053000507 learned_addr learned my addr 172.21.15.155:0/4053000507 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:32:28.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.976+0000 7f9a41427640 1 -- 172.21.15.155:0/4053000507 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 981457978 0 0) 0x7f9a3c142520 con 0x7f9a3c13f710 2024-03-25T18:32:28.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.976+0000 7f9a41427640 10 monclient(hunting): _negotiate 2024-03-25T18:32:28.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.976+0000 7f9a41427640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:28.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.976+0000 7f9a41427640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:28.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.976+0000 7f9a41427640 10 monclient(hunting): my global_id is 4572 2024-03-25T18:32:28.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.976+0000 7f9a41427640 1 -- 172.21.15.155:0/4053000507 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a28003520 con 0x7f9a3c13f710 2024-03-25T18:32:28.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.977+0000 7f9a41427640 1 -- 172.21.15.155:0/4053000507 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3269735148 0 0) 0x7f9a3c0b28a0 con 0x7f9a3c1430c0 2024-03-25T18:32:28.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.977+0000 7f9a41427640 10 monclient(hunting): _negotiate 2024-03-25T18:32:28.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.977+0000 7f9a41427640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:28.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.977+0000 7f9a41427640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:28.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.977+0000 7f9a41427640 10 monclient(hunting): my global_id is 4567 2024-03-25T18:32:28.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.977+0000 7f9a41427640 1 -- 172.21.15.155:0/4053000507 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a3c142520 con 0x7f9a3c1430c0 2024-03-25T18:32:28.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.984+0000 7f9a41427640 1 -- 172.21.15.155:0/4053000507 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2678641905 0 0) 0x7f9a28003520 con 0x7f9a3c13f710 2024-03-25T18:32:28.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.984+0000 7f9a41427640 1 -- 172.21.15.155:0/4053000507 >> v1:172.21.15.155:6790/0 conn(0x7f9a3c0a98f0 legacy=0x7f9a3c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:28.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.984+0000 7f9a41427640 1 -- 172.21.15.155:0/4053000507 >> v1:172.21.15.167:6789/0 conn(0x7f9a3c1430c0 legacy=0x7f9a3c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:28.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.984+0000 7f9a41427640 10 monclient: _finish_hunting 0 2024-03-25T18:32:28.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.984+0000 7f9a41427640 1 monclient: found mon.noname-a 2024-03-25T18:32:28.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.984+0000 7f9a41427640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:28.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.984+0000 7f9a41427640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-25T18:32:28.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.984+0000 7f9a41427640 1 -- 172.21.15.155:0/4053000507 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a3c0b44c0 con 0x7f9a3c13f710 2024-03-25T18:32:28.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.984+0000 7f9a41427640 10 monclient: _finish_auth 0 2024-03-25T18:32:28.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.984+0000 7f9a41427640 10 monclient: _check_auth_tickets 2024-03-25T18:32:28.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.984+0000 7f9a41427640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:28.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.984+0000 7f9a41427640 1 -- 172.21.15.155:0/4053000507 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f9a2c001690 con 0x7f9a3c13f710 2024-03-25T18:32:28.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.984+0000 7f9a41427640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:28.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.984+0000 7f9a41427640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:32:28.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.984+0000 7f9a41427640 10 monclient: dump: 2024-03-25T18:32:28.986 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:32:28.986 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:32:28.987 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:28.987 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:28.987 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:28.987 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:32:28.987 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:32:28.987 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:32:28.987 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:32:28.987 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:32:28.987 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.984+0000 7f9a4468d640 5 monclient: authenticate success, global_id 4572 2024-03-25T18:32:28.987 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.984+0000 7f9a4468d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:28.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.991+0000 7f9a41427640 1 -- 172.21.15.155:0/4053000507 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a2c0017f0 con 0x7f9a3c13f710 2024-03-25T18:32:28.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.991+0000 7f9a41427640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:28.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.991+0000 7f9a41427640 1 -- 172.21.15.155:0/4053000507 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f9a2c001c00 con 0x7f9a3c13f710 2024-03-25T18:32:28.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.991+0000 7f9a4468d640 10 monclient: get_monmap_and_config success 2024-03-25T18:32:28.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.991+0000 7f9a4468d640 4 set_mon_vals no callback set 2024-03-25T18:32:28.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.991+0000 7f9a41427640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:28.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.991+0000 7f9a41427640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:32:28.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.991+0000 7f9a41427640 10 monclient: dump: 2024-03-25T18:32:28.993 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:32:28.993 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:32:28.993 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:28.993 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:28.994 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:28.994 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:32:28.994 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:32:28.994 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:32:28.994 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:32:28.994 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:32:28.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.992+0000 7f9a4468d640 10 monclient: shutdown 2024-03-25T18:32:28.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.992+0000 7f9a4468d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:28.997 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.992+0000 7f9a4468d640 1 -- 172.21.15.155:0/4053000507 >> v1:172.21.15.155:6789/0 conn(0x7f9a3c13f710 legacy=0x7f9a3c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:28.997 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.992+0000 7f9a4468d640 1 -- 172.21.15.155:0/4053000507 shutdown_connections 2024-03-25T18:32:28.997 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.992+0000 7f9a4468d640 1 -- 172.21.15.155:0/4053000507 >> 172.21.15.155:0/4053000507 conn(0x7f9a3c0b7ea0 msgr2=0x7f9a3c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:28.997 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.992+0000 7f9a4468d640 1 -- 172.21.15.155:0/4053000507 shutdown_connections 2024-03-25T18:32:28.997 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.992+0000 7f9a4468d640 1 -- 172.21.15.155:0/4053000507 wait complete. 2024-03-25T18:32:28.997 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.993+0000 7f9a4468d640 10 monclient: build_initial_monmap 2024-03-25T18:32:28.997 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.993+0000 7f9a4468d640 10 monclient: monmap: 2024-03-25T18:32:28.997 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:32:28.997 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:32:28.997 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:32:28.997 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:32:28.997 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:28.997 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:32:28.998 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:32:28.998 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:32:28.998 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:32:28.998 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:32:28.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.993+0000 7f9a4468d640 1 librados: starting msgr at 2024-03-25T18:32:28.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.993+0000 7f9a4468d640 1 librados: starting objecter 2024-03-25T18:32:28.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.994+0000 7f9a4468d640 1 Processor -- start 2024-03-25T18:32:28.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.994+0000 7f9a4468d640 1 -- start start 2024-03-25T18:32:28.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.994+0000 7f9a4468d640 1 librados: setting wanted keys 2024-03-25T18:32:28.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.994+0000 7f9a4468d640 1 librados: calling monclient init 2024-03-25T18:32:28.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.994+0000 7f9a4468d640 10 monclient: init 2024-03-25T18:32:28.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.994+0000 7f9a4468d640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:28.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.994+0000 7f9a4468d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:32:28.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.994+0000 7f9a4468d640 10 monclient(hunting): picked mon.noname-a con 0x7f9a3c0a98f0 addr v1:172.21.15.155:6789/0 2024-03-25T18:32:28.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.994+0000 7f9a4468d640 10 monclient(hunting): picked mon.noname-b con 0x7f9a3c13f710 addr v1:172.21.15.167:6789/0 2024-03-25T18:32:28.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.994+0000 7f9a4468d640 10 monclient(hunting): picked mon.noname-c con 0x7f9a3c1430c0 addr v1:172.21.15.155:6790/0 2024-03-25T18:32:28.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.994+0000 7f9a4468d640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a3c0abd70 con 0x7f9a3c0a98f0 2024-03-25T18:32:28.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.994+0000 7f9a4468d640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a3c1d7710 con 0x7f9a3c13f710 2024-03-25T18:32:28.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.994+0000 7f9a4468d640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a3c1d8340 con 0x7f9a3c1430c0 2024-03-25T18:32:28.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.994+0000 7f9a4468d640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:28.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:28.994+0000 7f9a4468d640 10 monclient(hunting): authenticate will time out at 7995.304526s 2024-03-25T18:32:29.001 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.000+0000 7f9a42429640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f9a3c0a98f0 0x7f9a3c137be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:37548/0 (socket says 172.21.15.155:37548) 2024-03-25T18:32:29.001 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.000+0000 7f9a42429640 1 -- 172.21.15.155:0/1844007427 learned_addr learned my addr 172.21.15.155:0/1844007427 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:32:29.002 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.001+0000 7f9a32ffd640 10 client.?.objecter ms_handle_connect 0x7f9a3c0a98f0 2024-03-25T18:32:29.002 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.001+0000 7f9a32ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:29.010 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.009+0000 7f9a32ffd640 1 -- 172.21.15.155:0/1844007427 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1877113872 0 0) 0x7f9a3c0abd70 con 0x7f9a3c0a98f0 2024-03-25T18:32:29.010 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.009+0000 7f9a32ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:32:29.010 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.009+0000 7f9a32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:29.010 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.009+0000 7f9a32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:29.010 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.009+0000 7f9a32ffd640 10 monclient(hunting): my global_id is 4575 2024-03-25T18:32:29.010 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.009+0000 7f9a32ffd640 1 -- 172.21.15.155:0/1844007427 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a20004eb0 con 0x7f9a3c0a98f0 2024-03-25T18:32:29.012 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.011+0000 7f9a32ffd640 10 client.?.objecter ms_handle_connect 0x7f9a3c13f710 2024-03-25T18:32:29.012 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.011+0000 7f9a32ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:29.019 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.018+0000 7f9a32ffd640 10 client.?.objecter ms_handle_connect 0x7f9a3c1430c0 2024-03-25T18:32:29.019 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.018+0000 7f9a32ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:29.024 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.023+0000 7f9a32ffd640 1 -- 172.21.15.155:0/1844007427 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1272202999 0 0) 0x7f9a3c1d7710 con 0x7f9a3c13f710 2024-03-25T18:32:29.024 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.023+0000 7f9a32ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:32:29.024 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.023+0000 7f9a32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:29.024 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.023+0000 7f9a32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:29.025 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.023+0000 7f9a32ffd640 10 monclient(hunting): my global_id is 4570 2024-03-25T18:32:29.025 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.023+0000 7f9a32ffd640 1 -- 172.21.15.155:0/1844007427 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a3c0abd70 con 0x7f9a3c13f710 2024-03-25T18:32:29.027 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.026+0000 7f9a32ffd640 1 -- 172.21.15.155:0/1844007427 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2811695918 0 0) 0x7f9a20004eb0 con 0x7f9a3c0a98f0 2024-03-25T18:32:29.027 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.026+0000 7f9a32ffd640 1 -- 172.21.15.155:0/1844007427 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a3c1d7710 con 0x7f9a3c0a98f0 2024-03-25T18:32:29.027 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.026+0000 7f9a32ffd640 1 -- 172.21.15.155:0/1844007427 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f9a380019d0 con 0x7f9a3c0a98f0 2024-03-25T18:32:29.028 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.026+0000 7f9a32ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:29.028 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.026+0000 7f9a32ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:32:29.028 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.026+0000 7f9a32ffd640 10 monclient(hunting): dump: 2024-03-25T18:32:29.028 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:32:29.028 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:32:29.028 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:29.028 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:29.028 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:29.028 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:32:29.028 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:32:29.028 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:32:29.028 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:32:29.028 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:32:29.028 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.026+0000 7f9a32ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:32:29.029 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.028+0000 7f9a32ffd640 1 -- 172.21.15.155:0/1844007427 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2174372382 0 0) 0x7f9a3c1d8340 con 0x7f9a3c1430c0 2024-03-25T18:32:29.029 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.028+0000 7f9a32ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:32:29.029 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.028+0000 7f9a32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:29.029 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.028+0000 7f9a32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:29.029 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.028+0000 7f9a32ffd640 10 monclient(hunting): my global_id is 4583 2024-03-25T18:32:29.029 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.028+0000 7f9a32ffd640 1 -- 172.21.15.155:0/1844007427 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a20004eb0 con 0x7f9a3c1430c0 2024-03-25T18:32:29.044 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.042+0000 7f9a32ffd640 1 -- 172.21.15.155:0/1844007427 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 417726702 0 0) 0x7f9a3c1d7710 con 0x7f9a3c0a98f0 2024-03-25T18:32:29.044 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.042+0000 7f9a32ffd640 1 -- 172.21.15.155:0/1844007427 >> v1:172.21.15.155:6790/0 conn(0x7f9a3c1430c0 legacy=0x7f9a3c1d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:29.044 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.042+0000 7f9a32ffd640 1 -- 172.21.15.155:0/1844007427 >> v1:172.21.15.167:6789/0 conn(0x7f9a3c13f710 legacy=0x7f9a3c1d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:29.044 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.042+0000 7f9a32ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:32:29.044 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.042+0000 7f9a32ffd640 1 monclient: found mon.a 2024-03-25T18:32:29.044 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.042+0000 7f9a32ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:29.044 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.042+0000 7f9a32ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:32:29.045 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.042+0000 7f9a32ffd640 1 -- 172.21.15.155:0/1844007427 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a3c1d84d0 con 0x7f9a3c0a98f0 2024-03-25T18:32:29.045 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.042+0000 7f9a32ffd640 10 monclient: _finish_auth 0 2024-03-25T18:32:29.045 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.042+0000 7f9a32ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:32:29.045 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.042+0000 7f9a32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:29.045 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.042+0000 7f9a4468d640 5 monclient: authenticate success, global_id 4575 2024-03-25T18:32:29.045 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.042+0000 7f9a4468d640 10 monclient: _renew_subs 2024-03-25T18:32:29.045 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.042+0000 7f9a4468d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:32:29.045 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.042+0000 7f9a4468d640 1 -- 172.21.15.155:0/1844007427 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a3c1d41e0 con 0x7f9a3c0a98f0 2024-03-25T18:32:29.045 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.042+0000 7f9a4468d640 10 client.4575.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:32:29.045 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.042+0000 7f9a4468d640 10 monclient: _renew_subs 2024-03-25T18:32:29.045 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.042+0000 7f9a4468d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:32:29.045 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.042+0000 7f9a4468d640 1 -- 172.21.15.155:0/1844007427 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a3c1d5150 con 0x7f9a3c0a98f0 2024-03-25T18:32:29.045 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.042+0000 7f9a4468d640 1 librados: init done 2024-03-25T18:32:29.046 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.043+0000 7f9a4468d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:29.046 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.043+0000 7f9a4468d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:29.046 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.043+0000 7f9a4468d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:32:29.046 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.043+0000 7f9a4468d640 1 -- 172.21.15.155:0/1844007427 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a3c0627b0 con 0x7f9a3c0a98f0 2024-03-25T18:32:29.369 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.367+0000 7f9a32ffd640 1 -- 172.21.15.155:0/1844007427 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a38003610 con 0x7f9a3c0a98f0 2024-03-25T18:32:29.369 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.367+0000 7f9a32ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:29.369 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.367+0000 7f9a32ffd640 1 -- 172.21.15.155:0/1844007427 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f9a38003a20 con 0x7f9a3c0a98f0 2024-03-25T18:32:29.369 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.367+0000 7f9a32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:29.369 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.367+0000 7f9a32ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:32:29.369 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.367+0000 7f9a32ffd640 10 monclient: dump: 2024-03-25T18:32:29.369 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:32:29.369 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:32:29.369 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:29.369 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:29.370 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:29.370 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:32:29.370 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:32:29.370 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:32:29.370 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:32:29.370 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:32:29.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.367+0000 7f9a40c26640 4 set_mon_vals no callback set 2024-03-25T18:32:29.371 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.368+0000 7f9a32ffd640 1 -- 172.21.15.155:0/1844007427 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7f9a38003c40 con 0x7f9a3c0a98f0 2024-03-25T18:32:29.371 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.369+0000 7f9a41c28640 10 monclient: get_auth_request con 0x7f9a2006de20 auth_method 0 2024-03-25T18:32:29.371 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.369+0000 7f9a32ffd640 10 client.4575.objecter ms_handle_connect 0x7f9a2006de20 2024-03-25T18:32:29.379 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.378+0000 7f9a32ffd640 1 -- 172.21.15.155:0/1844007427 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7f9a380547d0 con 0x7f9a3c0a98f0 2024-03-25T18:32:29.379 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.378+0000 7f9a32ffd640 10 client.4575.objecter ms_dispatch 0x7f9a3c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:32:29.379 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.378+0000 7f9a32ffd640 3 client.4575.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:32:29.379 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.378+0000 7f9a32ffd640 3 client.4575.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:32:29.380 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.378+0000 7f9a32ffd640 20 client.4575.objecter dump_active .. 0 homeless 2024-03-25T18:32:29.380 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.378+0000 7f9a32ffd640 1 -- 172.21.15.155:0/1844007427 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9a380502e0 con 0x7f9a3c0a98f0 2024-03-25T18:32:29.380 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.378+0000 7f9a32ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:29.380 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.378+0000 7f9a32ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:32:29.600 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.598+0000 7f9a4468d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:29.600 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.598+0000 7f9a4468d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:29.600 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.598+0000 7f9a4468d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:32:29.600 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.598+0000 7f9a4468d640 1 -- 172.21.15.155:0/1844007427 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9a3c1d78a0 con 0x7f9a3c0a98f0 2024-03-25T18:32:29.629 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.628+0000 7f9a32ffd640 1 -- 172.21.15.155:0/1844007427 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7f9a38052210 con 0x7f9a3c0a98f0 2024-03-25T18:32:29.629 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.628+0000 7f9a32ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:29.630 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.628+0000 7f9a32ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:32:29.630 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:32:29.630 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:32:29.632 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.630+0000 7f9a4468d640 10 librados: watch_flush enter 2024-03-25T18:32:29.632 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.630+0000 7f9a4468d640 10 librados: watch_flush exit 2024-03-25T18:32:29.632 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.630+0000 7f9a4468d640 20 client.4575.objecter shutdown clearing up homeless session... 2024-03-25T18:32:29.632 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.630+0000 7f9a4468d640 10 client.4575.objecter successfully canceled tick 2024-03-25T18:32:29.632 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.630+0000 7f9a4468d640 1 -- 172.21.15.155:0/1844007427 >> v1:172.21.15.167:6800/2919070503 conn(0x7f9a2006de20 legacy=0x7f9a200702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:29.632 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.630+0000 7f9a4468d640 10 monclient: shutdown 2024-03-25T18:32:29.633 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.630+0000 7f9a4468d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:29.633 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.630+0000 7f9a4468d640 1 -- 172.21.15.155:0/1844007427 >> v1:172.21.15.155:6789/0 conn(0x7f9a3c0a98f0 legacy=0x7f9a3c137be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:29.633 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.630+0000 7f9a4468d640 1 -- 172.21.15.155:0/1844007427 shutdown_connections 2024-03-25T18:32:29.633 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.630+0000 7f9a4468d640 1 -- 172.21.15.155:0/1844007427 >> 172.21.15.155:0/1844007427 conn(0x7f9a3c0b7ea0 msgr2=0x7f9a3c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:29.633 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.630+0000 7f9a4468d640 1 -- 172.21.15.155:0/1844007427 shutdown_connections 2024-03-25T18:32:29.633 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.630+0000 7f9a4468d640 1 -- 172.21.15.155:0/1844007427 wait complete. 2024-03-25T18:32:29.633 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:29.630+0000 7f9a4468d640 1 librados: shutdown 2024-03-25T18:32:29.641 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:32:36.446 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:32:36.567 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.566+0000 7f4e24865640 10 monclient: get_monmap_and_config 2024-03-25T18:32:36.567 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.566+0000 7f4e24865640 10 monclient: build_initial_monmap 2024-03-25T18:32:36.567 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.566+0000 7f4e24865640 10 monclient: monmap: 2024-03-25T18:32:36.567 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:32:36.568 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:32:36.568 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:32:36.567316+0000 2024-03-25T18:32:36.568 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:32:36.567316+0000 2024-03-25T18:32:36.568 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:36.568 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:32:36.568 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:32:36.568 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:32:36.568 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:32:36.568 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:32:36.569 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.567+0000 7f4e24865640 1 Processor -- start 2024-03-25T18:32:36.569 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.567+0000 7f4e24865640 1 -- start start 2024-03-25T18:32:36.569 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.567+0000 7f4e24865640 10 monclient: init 2024-03-25T18:32:36.569 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.567+0000 7f4e24865640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:36.569 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.567+0000 7f4e24865640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:32:36.569 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.567+0000 7f4e24865640 10 monclient(hunting): picked mon.noname-b con 0x7f4e200a98f0 addr v1:172.21.15.155:6790/0 2024-03-25T18:32:36.569 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.567+0000 7f4e24865640 10 monclient(hunting): picked mon.noname-c con 0x7f4e2013f710 addr v1:172.21.15.167:6789/0 2024-03-25T18:32:36.570 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.567+0000 7f4e24865640 10 monclient(hunting): picked mon.noname-a con 0x7f4e201430c0 addr v1:172.21.15.155:6789/0 2024-03-25T18:32:36.570 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.567+0000 7f4e24865640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e20142520 con 0x7f4e201430c0 2024-03-25T18:32:36.570 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.567+0000 7f4e24865640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e200b28a0 con 0x7f4e2013f710 2024-03-25T18:32:36.570 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.568+0000 7f4e24865640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e200b2a30 con 0x7f4e200a98f0 2024-03-25T18:32:36.570 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.568+0000 7f4e24865640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:36.570 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.568+0000 7f4e24865640 10 monclient(hunting): authenticate will time out at 8002.877768s 2024-03-25T18:32:36.574 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.573+0000 7f4e1e59c640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f4e200a98f0 0x7f4e200a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:42034/0 (socket says 172.21.15.155:42034) 2024-03-25T18:32:36.574 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.573+0000 7f4e1e59c640 1 -- 172.21.15.155:0/4076762459 learned_addr learned my addr 172.21.15.155:0/4076762459 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:32:36.589 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.588+0000 7f4e1d59a640 1 -- 172.21.15.155:0/4076762459 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2378422110 0 0) 0x7f4e200b2a30 con 0x7f4e200a98f0 2024-03-25T18:32:36.589 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.588+0000 7f4e1d59a640 10 monclient(hunting): _negotiate 2024-03-25T18:32:36.589 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.588+0000 7f4e1d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:36.589 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.588+0000 7f4e1d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:36.589 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.588+0000 7f4e1d59a640 10 monclient(hunting): my global_id is 4586 2024-03-25T18:32:36.589 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.588+0000 7f4e1d59a640 1 -- 172.21.15.155:0/4076762459 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e08003520 con 0x7f4e200a98f0 2024-03-25T18:32:36.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.592+0000 7f4e1d59a640 1 -- 172.21.15.155:0/4076762459 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3840328235 0 0) 0x7f4e20142520 con 0x7f4e201430c0 2024-03-25T18:32:36.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.592+0000 7f4e1d59a640 10 monclient(hunting): _negotiate 2024-03-25T18:32:36.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.592+0000 7f4e1d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:36.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.592+0000 7f4e1d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:36.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.592+0000 7f4e1d59a640 10 monclient(hunting): my global_id is 4578 2024-03-25T18:32:36.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.592+0000 7f4e1d59a640 1 -- 172.21.15.155:0/4076762459 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e200b2a30 con 0x7f4e201430c0 2024-03-25T18:32:36.595 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.593+0000 7f4e1d59a640 1 -- 172.21.15.155:0/4076762459 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2905530805 0 0) 0x7f4e200b28a0 con 0x7f4e2013f710 2024-03-25T18:32:36.595 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.593+0000 7f4e1d59a640 10 monclient(hunting): _negotiate 2024-03-25T18:32:36.595 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.593+0000 7f4e1d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:36.595 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.593+0000 7f4e1d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:36.595 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.593+0000 7f4e1d59a640 10 monclient(hunting): my global_id is 4573 2024-03-25T18:32:36.595 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.593+0000 7f4e1d59a640 1 -- 172.21.15.155:0/4076762459 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e20142520 con 0x7f4e2013f710 2024-03-25T18:32:36.603 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.601+0000 7f4e1d59a640 1 -- 172.21.15.155:0/4076762459 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2760095327 0 0) 0x7f4e08003520 con 0x7f4e200a98f0 2024-03-25T18:32:36.603 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.601+0000 7f4e1d59a640 1 -- 172.21.15.155:0/4076762459 >> v1:172.21.15.167:6789/0 conn(0x7f4e2013f710 legacy=0x7f4e201429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:36.603 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.601+0000 7f4e1d59a640 1 -- 172.21.15.155:0/4076762459 >> v1:172.21.15.155:6789/0 conn(0x7f4e201430c0 legacy=0x7f4e2014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:36.603 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.601+0000 7f4e1d59a640 10 monclient: _finish_hunting 0 2024-03-25T18:32:36.603 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.601+0000 7f4e1d59a640 1 monclient: found mon.noname-b 2024-03-25T18:32:36.603 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.601+0000 7f4e1d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:36.603 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.602+0000 7f4e1d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-25T18:32:36.603 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.602+0000 7f4e1d59a640 1 -- 172.21.15.155:0/4076762459 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e200b44c0 con 0x7f4e200a98f0 2024-03-25T18:32:36.604 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.602+0000 7f4e1d59a640 10 monclient: _finish_auth 0 2024-03-25T18:32:36.604 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.602+0000 7f4e1d59a640 10 monclient: _check_auth_tickets 2024-03-25T18:32:36.604 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.602+0000 7f4e1d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:36.604 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.602+0000 7f4e1d59a640 1 -- 172.21.15.155:0/4076762459 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f4e10001690 con 0x7f4e200a98f0 2024-03-25T18:32:36.604 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.602+0000 7f4e24865640 5 monclient: authenticate success, global_id 4586 2024-03-25T18:32:36.604 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.602+0000 7f4e24865640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:36.604 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.602+0000 7f4e1d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:36.604 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.602+0000 7f4e1d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:32:36.604 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.602+0000 7f4e1d59a640 10 monclient: dump: 2024-03-25T18:32:36.604 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:32:36.604 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:32:36.604 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:36.604 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:36.604 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:36.604 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:32:36.604 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:32:36.605 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:32:36.605 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:32:36.605 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:32:36.605 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.602+0000 7f4e24865640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:36.610 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.609+0000 7f4e1d59a640 1 -- 172.21.15.155:0/4076762459 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e100017f0 con 0x7f4e200a98f0 2024-03-25T18:32:36.610 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.609+0000 7f4e1d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:36.610 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.609+0000 7f4e1d59a640 1 -- 172.21.15.155:0/4076762459 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f4e10001c00 con 0x7f4e200a98f0 2024-03-25T18:32:36.610 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.609+0000 7f4e1d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:36.610 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.609+0000 7f4e1d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:32:36.610 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.609+0000 7f4e1d59a640 10 monclient: dump: 2024-03-25T18:32:36.610 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:32:36.610 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:32:36.610 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:36.611 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:36.611 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:36.611 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:32:36.611 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:32:36.611 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:32:36.611 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:32:36.611 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:32:36.611 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.609+0000 7f4e24865640 10 monclient: get_monmap_and_config success 2024-03-25T18:32:36.611 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.609+0000 7f4e24865640 4 set_mon_vals no callback set 2024-03-25T18:32:36.612 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.609+0000 7f4e24865640 10 monclient: shutdown 2024-03-25T18:32:36.612 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.609+0000 7f4e24865640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:36.612 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.609+0000 7f4e24865640 1 -- 172.21.15.155:0/4076762459 >> v1:172.21.15.155:6790/0 conn(0x7f4e200a98f0 legacy=0x7f4e200a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:36.612 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.610+0000 7f4e24865640 1 -- 172.21.15.155:0/4076762459 shutdown_connections 2024-03-25T18:32:36.612 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.610+0000 7f4e24865640 1 -- 172.21.15.155:0/4076762459 >> 172.21.15.155:0/4076762459 conn(0x7f4e200b7ea0 msgr2=0x7f4e200b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:36.612 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.610+0000 7f4e24865640 1 -- 172.21.15.155:0/4076762459 shutdown_connections 2024-03-25T18:32:36.612 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.610+0000 7f4e24865640 1 -- 172.21.15.155:0/4076762459 wait complete. 2024-03-25T18:32:36.612 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.611+0000 7f4e24865640 10 monclient: build_initial_monmap 2024-03-25T18:32:36.613 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.611+0000 7f4e24865640 10 monclient: monmap: 2024-03-25T18:32:36.613 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:32:36.613 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:32:36.613 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:32:36.613 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:32:36.613 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:36.613 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:32:36.613 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:32:36.613 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:32:36.613 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:32:36.613 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:32:36.614 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.611+0000 7f4e24865640 1 librados: starting msgr at 2024-03-25T18:32:36.614 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.611+0000 7f4e24865640 1 librados: starting objecter 2024-03-25T18:32:36.614 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.611+0000 7f4e24865640 1 Processor -- start 2024-03-25T18:32:36.614 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.611+0000 7f4e24865640 1 -- start start 2024-03-25T18:32:36.615 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.611+0000 7f4e24865640 1 librados: setting wanted keys 2024-03-25T18:32:36.615 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.611+0000 7f4e24865640 1 librados: calling monclient init 2024-03-25T18:32:36.615 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.611+0000 7f4e24865640 10 monclient: init 2024-03-25T18:32:36.615 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.612+0000 7f4e24865640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:36.615 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.612+0000 7f4e24865640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:32:36.615 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.612+0000 7f4e24865640 10 monclient(hunting): picked mon.noname-c con 0x7f4e200a98f0 addr v1:172.21.15.155:6790/0 2024-03-25T18:32:36.615 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.612+0000 7f4e24865640 10 monclient(hunting): picked mon.noname-a con 0x7f4e2013f710 addr v1:172.21.15.155:6789/0 2024-03-25T18:32:36.615 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.612+0000 7f4e24865640 10 monclient(hunting): picked mon.noname-b con 0x7f4e201430c0 addr v1:172.21.15.167:6789/0 2024-03-25T18:32:36.615 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.612+0000 7f4e24865640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e20138b20 con 0x7f4e2013f710 2024-03-25T18:32:36.615 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.612+0000 7f4e24865640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e20137a80 con 0x7f4e201430c0 2024-03-25T18:32:36.615 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.612+0000 7f4e24865640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e20139880 con 0x7f4e200a98f0 2024-03-25T18:32:36.615 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.612+0000 7f4e24865640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:36.615 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.612+0000 7f4e24865640 10 monclient(hunting): authenticate will time out at 8002.922431s 2024-03-25T18:32:36.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.621+0000 7f4e1dd9b640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f4e2013f710 0x7f4e20139170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:50782/0 (socket says 172.21.15.155:50782) 2024-03-25T18:32:36.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.621+0000 7f4e1dd9b640 1 -- 172.21.15.155:0/1580554063 learned_addr learned my addr 172.21.15.155:0/1580554063 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:32:36.623 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.622+0000 7f4e0effd640 10 client.?.objecter ms_handle_connect 0x7f4e2013f710 2024-03-25T18:32:36.623 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.622+0000 7f4e0effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:36.628 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.627+0000 7f4e0effd640 10 client.?.objecter ms_handle_connect 0x7f4e200a98f0 2024-03-25T18:32:36.628 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.627+0000 7f4e0effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:36.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.636+0000 7f4e0effd640 1 -- 172.21.15.155:0/1580554063 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1621320832 0 0) 0x7f4e20138b20 con 0x7f4e2013f710 2024-03-25T18:32:36.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.636+0000 7f4e0effd640 10 monclient(hunting): _negotiate 2024-03-25T18:32:36.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.636+0000 7f4e0effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:36.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.636+0000 7f4e0effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:36.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.636+0000 7f4e0effd640 10 monclient(hunting): my global_id is 4581 2024-03-25T18:32:36.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.636+0000 7f4e0effd640 1 -- 172.21.15.155:0/1580554063 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e00004eb0 con 0x7f4e2013f710 2024-03-25T18:32:36.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.637+0000 7f4e0effd640 10 client.?.objecter ms_handle_connect 0x7f4e201430c0 2024-03-25T18:32:36.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.637+0000 7f4e0effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:36.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.644+0000 7f4e0effd640 1 -- 172.21.15.155:0/1580554063 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2494864375 0 0) 0x7f4e20139880 con 0x7f4e200a98f0 2024-03-25T18:32:36.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.644+0000 7f4e0effd640 10 monclient(hunting): _negotiate 2024-03-25T18:32:36.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.644+0000 7f4e0effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:36.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.644+0000 7f4e0effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:36.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.644+0000 7f4e0effd640 10 monclient(hunting): my global_id is 4589 2024-03-25T18:32:36.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.644+0000 7f4e0effd640 1 -- 172.21.15.155:0/1580554063 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e20138b20 con 0x7f4e200a98f0 2024-03-25T18:32:36.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.648+0000 7f4e0effd640 1 -- 172.21.15.155:0/1580554063 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1147369457 0 0) 0x7f4e00004eb0 con 0x7f4e2013f710 2024-03-25T18:32:36.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.648+0000 7f4e0effd640 1 -- 172.21.15.155:0/1580554063 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4e20139880 con 0x7f4e2013f710 2024-03-25T18:32:36.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.648+0000 7f4e0effd640 1 -- 172.21.15.155:0/1580554063 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f4e14001a10 con 0x7f4e2013f710 2024-03-25T18:32:36.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.648+0000 7f4e0effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:36.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.648+0000 7f4e0effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:32:36.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.648+0000 7f4e0effd640 10 monclient(hunting): dump: 2024-03-25T18:32:36.650 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:32:36.650 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:32:36.650 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:36.650 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:36.650 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:36.650 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:32:36.650 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:32:36.650 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:32:36.650 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:32:36.650 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:32:36.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.648+0000 7f4e0effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:32:36.654 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.653+0000 7f4e0effd640 1 -- 172.21.15.155:0/1580554063 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 542635809 0 0) 0x7f4e20137a80 con 0x7f4e201430c0 2024-03-25T18:32:36.654 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.653+0000 7f4e0effd640 10 monclient(hunting): _negotiate 2024-03-25T18:32:36.654 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.653+0000 7f4e0effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:36.654 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.653+0000 7f4e0effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:36.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.653+0000 7f4e0effd640 10 monclient(hunting): my global_id is 4576 2024-03-25T18:32:36.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.653+0000 7f4e0effd640 1 -- 172.21.15.155:0/1580554063 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e00004eb0 con 0x7f4e201430c0 2024-03-25T18:32:36.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.660+0000 7f4e0effd640 1 -- 172.21.15.155:0/1580554063 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1102617161 0 0) 0x7f4e20138b20 con 0x7f4e200a98f0 2024-03-25T18:32:36.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.660+0000 7f4e0effd640 1 -- 172.21.15.155:0/1580554063 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4e20137a80 con 0x7f4e200a98f0 2024-03-25T18:32:36.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.660+0000 7f4e0effd640 1 -- 172.21.15.155:0/1580554063 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f4e10001b00 con 0x7f4e200a98f0 2024-03-25T18:32:36.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.660+0000 7f4e0effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:36.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.660+0000 7f4e0effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:32:36.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.660+0000 7f4e0effd640 10 monclient(hunting): dump: 2024-03-25T18:32:36.662 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:32:36.662 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:32:36.662 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:36.662 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:36.662 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:36.662 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:32:36.662 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:32:36.662 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:32:36.662 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:32:36.663 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:32:36.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.666+0000 7f4e0effd640 1 -- 172.21.15.155:0/1580554063 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1061256399 0 0) 0x7f4e20139880 con 0x7f4e2013f710 2024-03-25T18:32:36.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.666+0000 7f4e0effd640 1 -- 172.21.15.155:0/1580554063 >> v1:172.21.15.155:6790/0 conn(0x7f4e200a98f0 legacy=0x7f4e20137370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:36.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.666+0000 7f4e0effd640 1 -- 172.21.15.155:0/1580554063 >> v1:172.21.15.167:6789/0 conn(0x7f4e201430c0 legacy=0x7f4e201e2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:36.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.666+0000 7f4e0effd640 10 monclient: _finish_hunting 0 2024-03-25T18:32:36.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.666+0000 7f4e0effd640 1 monclient: found mon.a 2024-03-25T18:32:36.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.666+0000 7f4e0effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:36.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.666+0000 7f4e0effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:32:36.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.666+0000 7f4e0effd640 1 -- 172.21.15.155:0/1580554063 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e20139a10 con 0x7f4e2013f710 2024-03-25T18:32:36.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.666+0000 7f4e0effd640 10 monclient: _finish_auth 0 2024-03-25T18:32:36.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.666+0000 7f4e0effd640 10 monclient: _check_auth_tickets 2024-03-25T18:32:36.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.666+0000 7f4e0effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:36.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.666+0000 7f4e24865640 5 monclient: authenticate success, global_id 4581 2024-03-25T18:32:36.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.666+0000 7f4e24865640 10 monclient: _renew_subs 2024-03-25T18:32:36.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.666+0000 7f4e24865640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:32:36.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.666+0000 7f4e24865640 1 -- 172.21.15.155:0/1580554063 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e201dab00 con 0x7f4e2013f710 2024-03-25T18:32:36.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.666+0000 7f4e24865640 10 client.4581.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:32:36.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.666+0000 7f4e24865640 10 monclient: _renew_subs 2024-03-25T18:32:36.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.666+0000 7f4e24865640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:32:36.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.666+0000 7f4e24865640 1 -- 172.21.15.155:0/1580554063 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e2013b4f0 con 0x7f4e2013f710 2024-03-25T18:32:36.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.666+0000 7f4e24865640 1 librados: init done 2024-03-25T18:32:36.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.667+0000 7f4e24865640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:36.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.667+0000 7f4e24865640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:36.670 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.667+0000 7f4e24865640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:32:36.670 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.667+0000 7f4e24865640 1 -- 172.21.15.155:0/1580554063 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e200b3e80 con 0x7f4e2013f710 2024-03-25T18:32:36.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.986+0000 7f4e0effd640 1 -- 172.21.15.155:0/1580554063 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e14001c70 con 0x7f4e2013f710 2024-03-25T18:32:36.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.986+0000 7f4e0effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:36.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.986+0000 7f4e0effd640 1 -- 172.21.15.155:0/1580554063 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f4e14003a60 con 0x7f4e2013f710 2024-03-25T18:32:36.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.986+0000 7f4e0effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:36.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.986+0000 7f4e0effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:32:36.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.986+0000 7f4e0effd640 10 monclient: dump: 2024-03-25T18:32:36.988 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:32:36.988 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:32:36.988 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:36.989 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:36.989 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:36.989 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:32:36.989 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:32:36.989 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:32:36.989 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:32:36.989 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:32:36.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.986+0000 7f4e1cd99640 4 set_mon_vals no callback set 2024-03-25T18:32:36.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.987+0000 7f4e0effd640 1 -- 172.21.15.155:0/1580554063 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7f4e14003c80 con 0x7f4e2013f710 2024-03-25T18:32:36.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.988+0000 7f4e1e59c640 10 monclient: get_auth_request con 0x7f4e0006dfb0 auth_method 0 2024-03-25T18:32:36.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.988+0000 7f4e0effd640 10 client.4581.objecter ms_handle_connect 0x7f4e0006dfb0 2024-03-25T18:32:36.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.993+0000 7f4e0effd640 1 -- 172.21.15.155:0/1580554063 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7f4e14054860 con 0x7f4e2013f710 2024-03-25T18:32:36.995 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.993+0000 7f4e0effd640 10 client.4581.objecter ms_dispatch 0x7f4e20000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:32:36.995 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.993+0000 7f4e0effd640 3 client.4581.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:32:36.995 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.993+0000 7f4e0effd640 3 client.4581.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:32:36.995 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.993+0000 7f4e0effd640 20 client.4581.objecter dump_active .. 0 homeless 2024-03-25T18:32:36.995 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.993+0000 7f4e0effd640 1 -- 172.21.15.155:0/1580554063 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4e14050370 con 0x7f4e2013f710 2024-03-25T18:32:36.995 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.993+0000 7f4e0effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:36.995 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:36.993+0000 7f4e0effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:32:37.226 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:37.224+0000 7f4e24865640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:37.226 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:37.224+0000 7f4e24865640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:37.226 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:37.224+0000 7f4e24865640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:32:37.226 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:37.224+0000 7f4e24865640 1 -- 172.21.15.155:0/1580554063 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4e200627b0 con 0x7f4e2013f710 2024-03-25T18:32:37.253 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:37.251+0000 7f4e0effd640 1 -- 172.21.15.155:0/1580554063 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7f4e140522a0 con 0x7f4e2013f710 2024-03-25T18:32:37.253 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:37.251+0000 7f4e0effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:37.253 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:37.251+0000 7f4e0effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:32:37.253 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:32:37.253 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:32:37.256 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:37.253+0000 7f4e24865640 10 librados: watch_flush enter 2024-03-25T18:32:37.256 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:37.254+0000 7f4e24865640 10 librados: watch_flush exit 2024-03-25T18:32:37.256 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:37.254+0000 7f4e24865640 20 client.4581.objecter shutdown clearing up homeless session... 2024-03-25T18:32:37.256 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:37.254+0000 7f4e24865640 10 client.4581.objecter successfully canceled tick 2024-03-25T18:32:37.256 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:37.254+0000 7f4e24865640 1 -- 172.21.15.155:0/1580554063 >> v1:172.21.15.167:6800/2919070503 conn(0x7f4e0006dfb0 legacy=0x7f4e00070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:37.256 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:37.254+0000 7f4e24865640 10 monclient: shutdown 2024-03-25T18:32:37.256 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:37.254+0000 7f4e24865640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:37.256 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:37.254+0000 7f4e24865640 1 -- 172.21.15.155:0/1580554063 >> v1:172.21.15.155:6789/0 conn(0x7f4e2013f710 legacy=0x7f4e20139170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:37.256 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:37.254+0000 7f4e24865640 1 -- 172.21.15.155:0/1580554063 shutdown_connections 2024-03-25T18:32:37.256 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:37.254+0000 7f4e24865640 1 -- 172.21.15.155:0/1580554063 >> 172.21.15.155:0/1580554063 conn(0x7f4e200b7ea0 msgr2=0x7f4e20074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:37.256 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:37.254+0000 7f4e24865640 1 -- 172.21.15.155:0/1580554063 shutdown_connections 2024-03-25T18:32:37.256 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:37.254+0000 7f4e24865640 1 -- 172.21.15.155:0/1580554063 wait complete. 2024-03-25T18:32:37.257 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:37.254+0000 7f4e24865640 1 librados: shutdown 2024-03-25T18:32:37.265 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:32:44.073 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:32:44.187 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.186+0000 7f54582c5640 10 monclient: get_monmap_and_config 2024-03-25T18:32:44.187 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.186+0000 7f54582c5640 10 monclient: build_initial_monmap 2024-03-25T18:32:44.187 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.186+0000 7f54582c5640 10 monclient: monmap: 2024-03-25T18:32:44.187 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:32:44.188 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:32:44.188 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:32:44.187010+0000 2024-03-25T18:32:44.188 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:32:44.187010+0000 2024-03-25T18:32:44.188 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:44.188 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:32:44.188 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:32:44.188 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:32:44.188 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:32:44.188 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:32:44.189 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.186+0000 7f54582c5640 1 Processor -- start 2024-03-25T18:32:44.189 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.186+0000 7f54582c5640 1 -- start start 2024-03-25T18:32:44.189 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.186+0000 7f54582c5640 10 monclient: init 2024-03-25T18:32:44.189 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.187+0000 7f54582c5640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:44.189 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.187+0000 7f54582c5640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:32:44.189 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.187+0000 7f54582c5640 10 monclient(hunting): picked mon.noname-c con 0x7f54500a98f0 addr v1:172.21.15.167:6789/0 2024-03-25T18:32:44.189 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.187+0000 7f54582c5640 10 monclient(hunting): picked mon.noname-a con 0x7f545013f710 addr v1:172.21.15.155:6789/0 2024-03-25T18:32:44.190 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.187+0000 7f54582c5640 10 monclient(hunting): picked mon.noname-b con 0x7f54501430c0 addr v1:172.21.15.155:6790/0 2024-03-25T18:32:44.190 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.187+0000 7f54582c5640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5450142520 con 0x7f545013f710 2024-03-25T18:32:44.190 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.187+0000 7f54582c5640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54500b28a0 con 0x7f54500a98f0 2024-03-25T18:32:44.190 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.187+0000 7f54582c5640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54500b2a30 con 0x7f54501430c0 2024-03-25T18:32:44.190 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.187+0000 7f54582c5640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:44.190 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.187+0000 7f54582c5640 10 monclient(hunting): authenticate will time out at 8010.497608s 2024-03-25T18:32:44.205 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.204+0000 7f5456061640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f54500a98f0 0x7f54500a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:42406/0 (socket says 172.21.15.155:42406) 2024-03-25T18:32:44.205 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.204+0000 7f5456061640 1 -- 172.21.15.155:0/1474769582 learned_addr learned my addr 172.21.15.155:0/1474769582 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:32:44.214 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.213+0000 7f545505f640 1 -- 172.21.15.155:0/1474769582 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3253282781 0 0) 0x7f54500b28a0 con 0x7f54500a98f0 2024-03-25T18:32:44.214 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.213+0000 7f545505f640 10 monclient(hunting): _negotiate 2024-03-25T18:32:44.214 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.213+0000 7f545505f640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:44.214 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.213+0000 7f545505f640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:44.214 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.213+0000 7f545505f640 10 monclient(hunting): my global_id is 4579 2024-03-25T18:32:44.214 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.213+0000 7f545505f640 1 -- 172.21.15.155:0/1474769582 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f543c003520 con 0x7f54500a98f0 2024-03-25T18:32:44.226 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.225+0000 7f545505f640 1 -- 172.21.15.155:0/1474769582 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2541440649 0 0) 0x7f543c003520 con 0x7f54500a98f0 2024-03-25T18:32:44.226 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.225+0000 7f545505f640 1 -- 172.21.15.155:0/1474769582 >> v1:172.21.15.155:6790/0 conn(0x7f54501430c0 legacy=0x7f545014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:44.226 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.225+0000 7f545505f640 1 -- 172.21.15.155:0/1474769582 >> v1:172.21.15.155:6789/0 conn(0x7f545013f710 legacy=0x7f54501429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:44.226 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.225+0000 7f545505f640 10 monclient: _finish_hunting 0 2024-03-25T18:32:44.226 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.225+0000 7f545505f640 1 monclient: found mon.noname-c 2024-03-25T18:32:44.227 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.225+0000 7f545505f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:44.227 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.225+0000 7f545505f640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.167:6789/0 2024-03-25T18:32:44.227 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.225+0000 7f545505f640 1 -- 172.21.15.155:0/1474769582 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54500b44c0 con 0x7f54500a98f0 2024-03-25T18:32:44.227 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.225+0000 7f545505f640 10 monclient: _finish_auth 0 2024-03-25T18:32:44.227 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.225+0000 7f545505f640 10 monclient: _check_auth_tickets 2024-03-25T18:32:44.227 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.225+0000 7f545505f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:44.227 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.225+0000 7f545505f640 1 -- 172.21.15.155:0/1474769582 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f544c001690 con 0x7f54500a98f0 2024-03-25T18:32:44.227 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.225+0000 7f545505f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:44.227 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.225+0000 7f545505f640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:32:44.227 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.225+0000 7f545505f640 10 monclient: dump: 2024-03-25T18:32:44.227 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:32:44.227 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:32:44.227 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:44.227 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:44.228 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:44.228 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:32:44.228 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:32:44.228 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:32:44.228 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:32:44.228 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:32:44.228 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.225+0000 7f54582c5640 5 monclient: authenticate success, global_id 4579 2024-03-25T18:32:44.228 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.225+0000 7f54582c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:44.234 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.233+0000 7f545505f640 1 -- 172.21.15.155:0/1474769582 <== mon.1 v1:172.21.15.167:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f544c0017f0 con 0x7f54500a98f0 2024-03-25T18:32:44.234 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.233+0000 7f545505f640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:44.234 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.233+0000 7f545505f640 1 -- 172.21.15.155:0/1474769582 <== mon.1 v1:172.21.15.167:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f544c001c00 con 0x7f54500a98f0 2024-03-25T18:32:44.234 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.233+0000 7f545505f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:44.234 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.233+0000 7f545505f640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:32:44.234 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.233+0000 7f545505f640 10 monclient: dump: 2024-03-25T18:32:44.234 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:32:44.234 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:32:44.235 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:44.235 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:44.235 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:44.235 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:32:44.235 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:32:44.235 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:32:44.235 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:32:44.235 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:32:44.235 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.233+0000 7f54582c5640 10 monclient: get_monmap_and_config success 2024-03-25T18:32:44.235 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.233+0000 7f54582c5640 4 set_mon_vals no callback set 2024-03-25T18:32:44.237 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.234+0000 7f54582c5640 10 monclient: shutdown 2024-03-25T18:32:44.237 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.234+0000 7f54582c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:44.237 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.234+0000 7f54582c5640 1 -- 172.21.15.155:0/1474769582 >> v1:172.21.15.167:6789/0 conn(0x7f54500a98f0 legacy=0x7f54500a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:44.237 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.234+0000 7f54582c5640 1 -- 172.21.15.155:0/1474769582 shutdown_connections 2024-03-25T18:32:44.237 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.234+0000 7f54582c5640 1 -- 172.21.15.155:0/1474769582 >> 172.21.15.155:0/1474769582 conn(0x7f54500b7ea0 msgr2=0x7f54500b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:44.237 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.234+0000 7f54582c5640 1 -- 172.21.15.155:0/1474769582 shutdown_connections 2024-03-25T18:32:44.237 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.234+0000 7f54582c5640 1 -- 172.21.15.155:0/1474769582 wait complete. 2024-03-25T18:32:44.237 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.235+0000 7f54582c5640 10 monclient: build_initial_monmap 2024-03-25T18:32:44.237 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.235+0000 7f54582c5640 10 monclient: monmap: 2024-03-25T18:32:44.237 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:32:44.237 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:32:44.238 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:32:44.238 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:32:44.238 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:44.238 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:32:44.238 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:32:44.238 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:32:44.238 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:32:44.238 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:32:44.238 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.235+0000 7f54582c5640 1 librados: starting msgr at 2024-03-25T18:32:44.238 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.235+0000 7f54582c5640 1 librados: starting objecter 2024-03-25T18:32:44.238 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.235+0000 7f54582c5640 1 Processor -- start 2024-03-25T18:32:44.238 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.236+0000 7f54582c5640 1 -- start start 2024-03-25T18:32:44.238 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.236+0000 7f54582c5640 1 librados: setting wanted keys 2024-03-25T18:32:44.238 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.236+0000 7f54582c5640 1 librados: calling monclient init 2024-03-25T18:32:44.239 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.236+0000 7f54582c5640 10 monclient: init 2024-03-25T18:32:44.239 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.236+0000 7f54582c5640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:44.239 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.236+0000 7f54582c5640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:32:44.239 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.236+0000 7f54582c5640 10 monclient(hunting): picked mon.noname-b con 0x7f54500a98f0 addr v1:172.21.15.167:6789/0 2024-03-25T18:32:44.239 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.236+0000 7f54582c5640 10 monclient(hunting): picked mon.noname-c con 0x7f545013f710 addr v1:172.21.15.155:6790/0 2024-03-25T18:32:44.239 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.236+0000 7f54582c5640 10 monclient(hunting): picked mon.noname-a con 0x7f54501430c0 addr v1:172.21.15.155:6789/0 2024-03-25T18:32:44.240 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.236+0000 7f54582c5640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54501db7d0 con 0x7f54501430c0 2024-03-25T18:32:44.240 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.236+0000 7f54582c5640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54500b1420 con 0x7f54500a98f0 2024-03-25T18:32:44.240 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.236+0000 7f54582c5640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54500b15b0 con 0x7f545013f710 2024-03-25T18:32:44.240 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.236+0000 7f54582c5640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:44.240 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.236+0000 7f54582c5640 10 monclient(hunting): authenticate will time out at 8010.546515s 2024-03-25T18:32:44.249 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.248+0000 7f5456862640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f54501430c0 0x7f54501d4350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:55558/0 (socket says 172.21.15.155:55558) 2024-03-25T18:32:44.249 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.248+0000 7f5456862640 1 -- 172.21.15.155:0/3418944220 learned_addr learned my addr 172.21.15.155:0/3418944220 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:32:44.252 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.250+0000 7f5442ffd640 10 client.?.objecter ms_handle_connect 0x7f545013f710 2024-03-25T18:32:44.252 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.250+0000 7f5442ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:44.252 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.251+0000 7f5442ffd640 10 client.?.objecter ms_handle_connect 0x7f54501430c0 2024-03-25T18:32:44.252 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.251+0000 7f5442ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:44.257 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.256+0000 7f5442ffd640 10 client.?.objecter ms_handle_connect 0x7f54500a98f0 2024-03-25T18:32:44.257 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.256+0000 7f5442ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:44.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.260+0000 7f5442ffd640 1 -- 172.21.15.155:0/3418944220 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 978265730 0 0) 0x7f54500b15b0 con 0x7f545013f710 2024-03-25T18:32:44.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.260+0000 7f5442ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:32:44.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.260+0000 7f5442ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:44.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.260+0000 7f5442ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:44.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.260+0000 7f5442ffd640 10 monclient(hunting): my global_id is 4595 2024-03-25T18:32:44.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.260+0000 7f5442ffd640 1 -- 172.21.15.155:0/3418944220 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5434004eb0 con 0x7f545013f710 2024-03-25T18:32:44.271 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.270+0000 7f5442ffd640 1 -- 172.21.15.155:0/3418944220 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2725533882 0 0) 0x7f54501db7d0 con 0x7f54501430c0 2024-03-25T18:32:44.271 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.270+0000 7f5442ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:32:44.271 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.270+0000 7f5442ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:44.271 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.270+0000 7f5442ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:44.271 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.270+0000 7f5442ffd640 10 monclient(hunting): my global_id is 4587 2024-03-25T18:32:44.271 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.270+0000 7f5442ffd640 1 -- 172.21.15.155:0/3418944220 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54500b15b0 con 0x7f54501430c0 2024-03-25T18:32:44.272 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.271+0000 7f5442ffd640 1 -- 172.21.15.155:0/3418944220 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 868897188 0 0) 0x7f54500b1420 con 0x7f54500a98f0 2024-03-25T18:32:44.272 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.271+0000 7f5442ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:32:44.272 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.271+0000 7f5442ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:44.272 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.271+0000 7f5442ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:44.272 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.271+0000 7f5442ffd640 10 monclient(hunting): my global_id is 4582 2024-03-25T18:32:44.272 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.271+0000 7f5442ffd640 1 -- 172.21.15.155:0/3418944220 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54501db7d0 con 0x7f54500a98f0 2024-03-25T18:32:44.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.286+0000 7f5442ffd640 1 -- 172.21.15.155:0/3418944220 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3411065376 0 0) 0x7f5434004eb0 con 0x7f545013f710 2024-03-25T18:32:44.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.286+0000 7f5442ffd640 1 -- 172.21.15.155:0/3418944220 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54500b1420 con 0x7f545013f710 2024-03-25T18:32:44.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.286+0000 7f5442ffd640 1 -- 172.21.15.155:0/3418944220 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f5444001930 con 0x7f545013f710 2024-03-25T18:32:44.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.286+0000 7f5442ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:44.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.286+0000 7f5442ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:32:44.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.286+0000 7f5442ffd640 10 monclient(hunting): dump: 2024-03-25T18:32:44.288 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:32:44.288 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:32:44.288 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:44.288 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:44.288 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:44.288 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:32:44.288 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:32:44.288 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:32:44.288 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:32:44.288 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:32:44.289 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.286+0000 7f5442ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:32:44.290 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.289+0000 7f5442ffd640 1 -- 172.21.15.155:0/3418944220 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 860951065 0 0) 0x7f54500b15b0 con 0x7f54501430c0 2024-03-25T18:32:44.291 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.289+0000 7f5442ffd640 1 -- 172.21.15.155:0/3418944220 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5434004eb0 con 0x7f54501430c0 2024-03-25T18:32:44.291 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.289+0000 7f5442ffd640 1 -- 172.21.15.155:0/3418944220 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f5448001c10 con 0x7f54501430c0 2024-03-25T18:32:44.291 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.289+0000 7f5442ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:44.291 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.289+0000 7f5442ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:32:44.291 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.289+0000 7f5442ffd640 10 monclient(hunting): dump: 2024-03-25T18:32:44.291 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:32:44.291 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:32:44.291 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:44.291 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:44.291 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:44.291 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:32:44.291 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:32:44.291 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:32:44.291 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:32:44.292 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:32:44.306 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.305+0000 7f5442ffd640 1 -- 172.21.15.155:0/3418944220 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4272726897 0 0) 0x7f54501db7d0 con 0x7f54500a98f0 2024-03-25T18:32:44.306 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.305+0000 7f5442ffd640 1 -- 172.21.15.155:0/3418944220 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54500b15b0 con 0x7f54500a98f0 2024-03-25T18:32:44.306 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.305+0000 7f5442ffd640 1 -- 172.21.15.155:0/3418944220 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f544c001b00 con 0x7f54500a98f0 2024-03-25T18:32:44.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.305+0000 7f5442ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:44.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.305+0000 7f5442ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:32:44.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.305+0000 7f5442ffd640 10 monclient(hunting): dump: 2024-03-25T18:32:44.307 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:32:44.307 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:32:44.307 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:44.307 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:44.307 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:44.307 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:32:44.307 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:32:44.307 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:32:44.307 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:32:44.307 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:32:44.309 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.308+0000 7f5442ffd640 1 -- 172.21.15.155:0/3418944220 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1299986999 0 0) 0x7f54500b1420 con 0x7f545013f710 2024-03-25T18:32:44.309 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.308+0000 7f5442ffd640 1 -- 172.21.15.155:0/3418944220 >> v1:172.21.15.167:6789/0 conn(0x7f54500a98f0 legacy=0x7f54501daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:44.310 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.308+0000 7f5442ffd640 1 -- 172.21.15.155:0/3418944220 >> v1:172.21.15.155:6789/0 conn(0x7f54501430c0 legacy=0x7f54501d4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:44.310 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.308+0000 7f5442ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:32:44.310 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.308+0000 7f5442ffd640 1 monclient: found mon.c 2024-03-25T18:32:44.310 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.308+0000 7f5442ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:44.310 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.308+0000 7f5442ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:32:44.310 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.308+0000 7f5442ffd640 1 -- 172.21.15.155:0/3418944220 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54500b1740 con 0x7f545013f710 2024-03-25T18:32:44.310 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.308+0000 7f5442ffd640 10 monclient: _finish_auth 0 2024-03-25T18:32:44.310 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.308+0000 7f5442ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:32:44.310 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.308+0000 7f5442ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:44.310 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.308+0000 7f54582c5640 5 monclient: authenticate success, global_id 4595 2024-03-25T18:32:44.310 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.308+0000 7f54582c5640 10 monclient: _renew_subs 2024-03-25T18:32:44.310 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.308+0000 7f54582c5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:32:44.310 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.308+0000 7f54582c5640 1 -- 172.21.15.155:0/3418944220 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54501db1a0 con 0x7f545013f710 2024-03-25T18:32:44.310 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.308+0000 7f54582c5640 10 client.4595.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:32:44.311 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.308+0000 7f54582c5640 10 monclient: _renew_subs 2024-03-25T18:32:44.311 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.308+0000 7f54582c5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:32:44.311 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.308+0000 7f54582c5640 1 -- 172.21.15.155:0/3418944220 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f54500b08f0 con 0x7f545013f710 2024-03-25T18:32:44.311 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.308+0000 7f54582c5640 1 librados: init done 2024-03-25T18:32:44.311 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.309+0000 7f54582c5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:44.311 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.309+0000 7f54582c5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:44.311 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.309+0000 7f54582c5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:32:44.312 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.309+0000 7f54582c5640 1 -- 172.21.15.155:0/3418944220 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54500627b0 con 0x7f545013f710 2024-03-25T18:32:44.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.638+0000 7f5442ffd640 1 -- 172.21.15.155:0/3418944220 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f54440035f0 con 0x7f545013f710 2024-03-25T18:32:44.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.638+0000 7f5442ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:44.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.638+0000 7f5442ffd640 1 -- 172.21.15.155:0/3418944220 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f5444003980 con 0x7f545013f710 2024-03-25T18:32:44.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.638+0000 7f5442ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:44.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.638+0000 7f5442ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:32:44.641 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.638+0000 7f5442ffd640 10 monclient: dump: 2024-03-25T18:32:44.641 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:32:44.641 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:32:44.641 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:44.641 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:44.641 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:44.641 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:32:44.641 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:32:44.641 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:32:44.641 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:32:44.641 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:32:44.641 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.638+0000 7f545485e640 4 set_mon_vals no callback set 2024-03-25T18:32:44.642 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.639+0000 7f5442ffd640 1 -- 172.21.15.155:0/3418944220 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7f5444003ba0 con 0x7f545013f710 2024-03-25T18:32:44.642 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.640+0000 7f5456061640 10 monclient: get_auth_request con 0x7f543406e110 auth_method 0 2024-03-25T18:32:44.642 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.640+0000 7f5442ffd640 10 client.4595.objecter ms_handle_connect 0x7f543406e110 2024-03-25T18:32:44.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.644+0000 7f5442ffd640 1 -- 172.21.15.155:0/3418944220 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7f54440535d0 con 0x7f545013f710 2024-03-25T18:32:44.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.644+0000 7f5442ffd640 10 client.4595.objecter ms_dispatch 0x7f5450000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:32:44.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.644+0000 7f5442ffd640 3 client.4595.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:32:44.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.644+0000 7f5442ffd640 3 client.4595.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:32:44.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.645+0000 7f5442ffd640 20 client.4595.objecter dump_active .. 0 homeless 2024-03-25T18:32:44.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.645+0000 7f5442ffd640 1 -- 172.21.15.155:0/3418944220 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f544404f0e0 con 0x7f545013f710 2024-03-25T18:32:44.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.645+0000 7f5442ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:44.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.645+0000 7f5442ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:32:44.898 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.896+0000 7f54582c5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:44.898 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.896+0000 7f54582c5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:44.898 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.896+0000 7f54582c5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:32:44.898 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.896+0000 7f54582c5640 1 -- 172.21.15.155:0/3418944220 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f54500abae0 con 0x7f545013f710 2024-03-25T18:32:44.956 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.954+0000 7f5442ffd640 1 -- 172.21.15.155:0/3418944220 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7f5444051010 con 0x7f545013f710 2024-03-25T18:32:44.956 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.954+0000 7f5442ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:44.956 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.954+0000 7f5442ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:32:44.956 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:32:44.956 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:32:44.959 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.957+0000 7f54582c5640 10 librados: watch_flush enter 2024-03-25T18:32:44.959 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.957+0000 7f54582c5640 10 librados: watch_flush exit 2024-03-25T18:32:44.959 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.957+0000 7f54582c5640 20 client.4595.objecter shutdown clearing up homeless session... 2024-03-25T18:32:44.959 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.957+0000 7f54582c5640 10 client.4595.objecter successfully canceled tick 2024-03-25T18:32:44.959 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.957+0000 7f54582c5640 1 -- 172.21.15.155:0/3418944220 >> v1:172.21.15.167:6800/2919070503 conn(0x7f543406e110 legacy=0x7f54340705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:44.959 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.957+0000 7f54582c5640 10 monclient: shutdown 2024-03-25T18:32:44.959 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.957+0000 7f54582c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:44.959 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.957+0000 7f54582c5640 1 -- 172.21.15.155:0/3418944220 >> v1:172.21.15.155:6790/0 conn(0x7f545013f710 legacy=0x7f54500b01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:44.959 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.957+0000 7f54582c5640 1 -- 172.21.15.155:0/3418944220 shutdown_connections 2024-03-25T18:32:44.960 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.957+0000 7f54582c5640 1 -- 172.21.15.155:0/3418944220 >> 172.21.15.155:0/3418944220 conn(0x7f54500b7ea0 msgr2=0x7f545013c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:44.960 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.957+0000 7f54582c5640 1 -- 172.21.15.155:0/3418944220 shutdown_connections 2024-03-25T18:32:44.960 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.957+0000 7f54582c5640 1 -- 172.21.15.155:0/3418944220 wait complete. 2024-03-25T18:32:44.960 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:44.957+0000 7f54582c5640 1 librados: shutdown 2024-03-25T18:32:44.971 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:32:47.236 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:08:20.840 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:47.248 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:08:20.852 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:47.285 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:08:20.889 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:47.293 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:08:20.872 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:47.297 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:08:20.876 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:47.301 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:08:20.905 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:47.306 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:08:20.911 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:47.321 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:08:20.900 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:47.332 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:08:20.937 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:51.778 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:32:51.895 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.894+0000 7ff5a5ed3640 10 monclient: get_monmap_and_config 2024-03-25T18:32:51.895 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.894+0000 7ff5a5ed3640 10 monclient: build_initial_monmap 2024-03-25T18:32:51.895 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.894+0000 7ff5a5ed3640 10 monclient: monmap: 2024-03-25T18:32:51.895 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:32:51.895 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:32:51.895 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:32:51.895018+0000 2024-03-25T18:32:51.896 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:32:51.895018+0000 2024-03-25T18:32:51.896 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:51.896 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:32:51.896 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:32:51.896 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:32:51.896 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:32:51.896 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:32:51.897 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.895+0000 7ff5a5ed3640 1 Processor -- start 2024-03-25T18:32:51.897 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.895+0000 7ff5a5ed3640 1 -- start start 2024-03-25T18:32:51.897 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.895+0000 7ff5a5ed3640 10 monclient: init 2024-03-25T18:32:51.897 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.895+0000 7ff5a5ed3640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:51.897 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.895+0000 7ff5a5ed3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:32:51.897 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.895+0000 7ff5a5ed3640 10 monclient(hunting): picked mon.noname-a con 0x7ff5a006ab80 addr v1:172.21.15.155:6789/0 2024-03-25T18:32:51.897 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.895+0000 7ff5a5ed3640 10 monclient(hunting): picked mon.noname-c con 0x7ff5a006af60 addr v1:172.21.15.167:6789/0 2024-03-25T18:32:51.897 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.895+0000 7ff5a5ed3640 10 monclient(hunting): picked mon.noname-b con 0x7ff5a00b00b0 addr v1:172.21.15.155:6790/0 2024-03-25T18:32:51.898 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.895+0000 7ff5a5ed3640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5a00b0500 con 0x7ff5a006ab80 2024-03-25T18:32:51.898 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.895+0000 7ff5a5ed3640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5a013ffa0 con 0x7ff5a006af60 2024-03-25T18:32:51.898 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.895+0000 7ff5a5ed3640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5a00af520 con 0x7ff5a00b00b0 2024-03-25T18:32:51.898 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.895+0000 7ff5a5ed3640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:51.898 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.895+0000 7ff5a5ed3640 10 monclient(hunting): authenticate will time out at 8018.205636s 2024-03-25T18:32:51.909 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.907+0000 7ff59effd640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7ff5a006af60 0x7ff5a013f890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:45030/0 (socket says 172.21.15.155:45030) 2024-03-25T18:32:51.909 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.907+0000 7ff59effd640 1 -- 172.21.15.155:0/4265984663 learned_addr learned my addr 172.21.15.155:0/4265984663 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:32:51.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.916+0000 7ff59e7fc640 1 -- 172.21.15.155:0/4265984663 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 243931 0 0) 0x7ff5a00b0500 con 0x7ff5a006ab80 2024-03-25T18:32:51.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.916+0000 7ff59e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:32:51.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.916+0000 7ff59e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:51.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.916+0000 7ff59e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:51.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.916+0000 7ff59e7fc640 10 monclient(hunting): my global_id is 4590 2024-03-25T18:32:51.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.916+0000 7ff59e7fc640 1 -- 172.21.15.155:0/4265984663 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff588003520 con 0x7ff5a006ab80 2024-03-25T18:32:51.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.922+0000 7ff59e7fc640 1 -- 172.21.15.155:0/4265984663 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1787902217 0 0) 0x7ff5a00af520 con 0x7ff5a00b00b0 2024-03-25T18:32:51.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.922+0000 7ff59e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:32:51.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.922+0000 7ff59e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:51.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.922+0000 7ff59e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:51.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.922+0000 7ff59e7fc640 10 monclient(hunting): my global_id is 4598 2024-03-25T18:32:51.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.922+0000 7ff59e7fc640 1 -- 172.21.15.155:0/4265984663 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5a00b0500 con 0x7ff5a00b00b0 2024-03-25T18:32:51.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.923+0000 7ff59e7fc640 1 -- 172.21.15.155:0/4265984663 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1704665812 0 0) 0x7ff5a013ffa0 con 0x7ff5a006af60 2024-03-25T18:32:51.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.923+0000 7ff59e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:32:51.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.923+0000 7ff59e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:51.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.923+0000 7ff59e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:51.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.923+0000 7ff59e7fc640 10 monclient(hunting): my global_id is 4585 2024-03-25T18:32:51.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.923+0000 7ff59e7fc640 1 -- 172.21.15.155:0/4265984663 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5a00af520 con 0x7ff5a006af60 2024-03-25T18:32:51.933 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.931+0000 7ff59e7fc640 1 -- 172.21.15.155:0/4265984663 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4021814369 0 0) 0x7ff588003520 con 0x7ff5a006ab80 2024-03-25T18:32:51.933 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.931+0000 7ff59e7fc640 1 -- 172.21.15.155:0/4265984663 >> v1:172.21.15.155:6790/0 conn(0x7ff5a00b00b0 legacy=0x7ff5a014dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:51.933 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.931+0000 7ff59e7fc640 1 -- 172.21.15.155:0/4265984663 >> v1:172.21.15.167:6789/0 conn(0x7ff5a006af60 legacy=0x7ff5a013f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:51.933 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.931+0000 7ff59e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:32:51.933 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.931+0000 7ff59e7fc640 1 monclient: found mon.noname-a 2024-03-25T18:32:51.933 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.931+0000 7ff59e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:51.933 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.931+0000 7ff59e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-25T18:32:51.933 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.931+0000 7ff59e7fc640 1 -- 172.21.15.155:0/4265984663 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5a0072890 con 0x7ff5a006ab80 2024-03-25T18:32:51.933 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.931+0000 7ff59e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:32:51.933 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.931+0000 7ff59e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:32:51.933 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.931+0000 7ff59e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:51.933 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.931+0000 7ff59e7fc640 1 -- 172.21.15.155:0/4265984663 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7ff590001690 con 0x7ff5a006ab80 2024-03-25T18:32:51.933 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.931+0000 7ff5a5ed3640 5 monclient: authenticate success, global_id 4590 2024-03-25T18:32:51.933 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.931+0000 7ff5a5ed3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:51.934 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.931+0000 7ff59e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:51.934 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.931+0000 7ff59e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:32:51.934 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.931+0000 7ff59e7fc640 10 monclient: dump: 2024-03-25T18:32:51.934 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:32:51.934 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:32:51.934 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:51.934 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:51.934 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:51.934 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:32:51.934 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:32:51.934 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:32:51.934 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:32:51.934 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:32:51.935 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.931+0000 7ff5a5ed3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:51.941 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.940+0000 7ff59e7fc640 1 -- 172.21.15.155:0/4265984663 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff5900017f0 con 0x7ff5a006ab80 2024-03-25T18:32:51.941 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.940+0000 7ff59e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:51.941 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.940+0000 7ff59e7fc640 1 -- 172.21.15.155:0/4265984663 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7ff590001c00 con 0x7ff5a006ab80 2024-03-25T18:32:51.941 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.940+0000 7ff5a5ed3640 10 monclient: get_monmap_and_config success 2024-03-25T18:32:51.941 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.940+0000 7ff5a5ed3640 4 set_mon_vals no callback set 2024-03-25T18:32:51.941 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.940+0000 7ff59e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:51.942 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.940+0000 7ff59e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:32:51.942 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.940+0000 7ff59e7fc640 10 monclient: dump: 2024-03-25T18:32:51.942 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:32:51.942 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:32:51.942 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:51.942 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:51.942 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:51.942 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:32:51.942 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:32:51.942 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:32:51.942 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:32:51.942 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:32:51.944 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.941+0000 7ff5a5ed3640 10 monclient: shutdown 2024-03-25T18:32:51.944 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.941+0000 7ff5a5ed3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:51.944 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.941+0000 7ff5a5ed3640 1 -- 172.21.15.155:0/4265984663 >> v1:172.21.15.155:6789/0 conn(0x7ff5a006ab80 legacy=0x7ff5a00ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:51.944 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.941+0000 7ff5a5ed3640 1 -- 172.21.15.155:0/4265984663 shutdown_connections 2024-03-25T18:32:51.945 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.941+0000 7ff5a5ed3640 1 -- 172.21.15.155:0/4265984663 >> 172.21.15.155:0/4265984663 conn(0x7ff5a00b4a80 msgr2=0x7ff5a00b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:51.945 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.941+0000 7ff5a5ed3640 1 -- 172.21.15.155:0/4265984663 shutdown_connections 2024-03-25T18:32:51.945 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.941+0000 7ff5a5ed3640 1 -- 172.21.15.155:0/4265984663 wait complete. 2024-03-25T18:32:51.945 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.942+0000 7ff5a5ed3640 10 monclient: build_initial_monmap 2024-03-25T18:32:51.945 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.942+0000 7ff5a5ed3640 10 monclient: monmap: 2024-03-25T18:32:51.945 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:32:51.945 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:32:51.945 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:32:51.945 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:32:51.945 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:51.945 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:32:51.945 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:32:51.945 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:32:51.945 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:32:51.945 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:32:51.946 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.942+0000 7ff5a5ed3640 1 librados: starting msgr at 2024-03-25T18:32:51.946 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.942+0000 7ff5a5ed3640 1 librados: starting objecter 2024-03-25T18:32:51.946 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.942+0000 7ff5a5ed3640 1 Processor -- start 2024-03-25T18:32:51.946 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.942+0000 7ff5a5ed3640 1 -- start start 2024-03-25T18:32:51.946 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.942+0000 7ff5a5ed3640 1 librados: setting wanted keys 2024-03-25T18:32:51.946 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.942+0000 7ff5a5ed3640 1 librados: calling monclient init 2024-03-25T18:32:51.946 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.942+0000 7ff5a5ed3640 10 monclient: init 2024-03-25T18:32:51.946 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.943+0000 7ff5a5ed3640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:51.946 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.943+0000 7ff5a5ed3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:32:51.946 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.943+0000 7ff5a5ed3640 10 monclient(hunting): picked mon.noname-c con 0x7ff5a006ab80 addr v1:172.21.15.155:6790/0 2024-03-25T18:32:51.946 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.943+0000 7ff5a5ed3640 10 monclient(hunting): picked mon.noname-b con 0x7ff5a00b00b0 addr v1:172.21.15.167:6789/0 2024-03-25T18:32:51.946 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.943+0000 7ff5a5ed3640 10 monclient(hunting): picked mon.noname-a con 0x7ff5a0140990 addr v1:172.21.15.155:6789/0 2024-03-25T18:32:51.946 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.943+0000 7ff5a5ed3640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5a0135820 con 0x7ff5a0140990 2024-03-25T18:32:51.946 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.943+0000 7ff5a5ed3640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5a0145710 con 0x7ff5a00b00b0 2024-03-25T18:32:51.947 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.943+0000 7ff5a5ed3640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5a01459f0 con 0x7ff5a006ab80 2024-03-25T18:32:51.947 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.943+0000 7ff5a5ed3640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:51.947 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.943+0000 7ff5a5ed3640 10 monclient(hunting): authenticate will time out at 8018.252968s 2024-03-25T18:32:51.961 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.959+0000 7ff59effd640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7ff5a00b00b0 0x7ff5a0134a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:45092/0 (socket says 172.21.15.155:45092) 2024-03-25T18:32:51.961 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.959+0000 7ff59effd640 1 -- 172.21.15.155:0/764475965 learned_addr learned my addr 172.21.15.155:0/764475965 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:32:51.963 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.962+0000 7ff587fff640 10 client.?.objecter ms_handle_connect 0x7ff5a00b00b0 2024-03-25T18:32:51.963 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.962+0000 7ff587fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:51.969 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.967+0000 7ff587fff640 1 -- 172.21.15.155:0/764475965 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 34003347 0 0) 0x7ff5a0145710 con 0x7ff5a00b00b0 2024-03-25T18:32:51.969 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.967+0000 7ff587fff640 10 monclient(hunting): _negotiate 2024-03-25T18:32:51.969 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.967+0000 7ff587fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:51.969 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.967+0000 7ff587fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:51.969 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.967+0000 7ff587fff640 10 monclient(hunting): my global_id is 4588 2024-03-25T18:32:51.969 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.967+0000 7ff587fff640 1 -- 172.21.15.155:0/764475965 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff58c004eb0 con 0x7ff5a00b00b0 2024-03-25T18:32:51.969 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.968+0000 7ff587fff640 10 client.?.objecter ms_handle_connect 0x7ff5a0140990 2024-03-25T18:32:51.969 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.968+0000 7ff587fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:51.971 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.970+0000 7ff587fff640 10 client.?.objecter ms_handle_connect 0x7ff5a006ab80 2024-03-25T18:32:51.972 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.970+0000 7ff587fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:51.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.977+0000 7ff587fff640 1 -- 172.21.15.155:0/764475965 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2888172386 0 0) 0x7ff5a0135820 con 0x7ff5a0140990 2024-03-25T18:32:51.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.977+0000 7ff587fff640 10 monclient(hunting): _negotiate 2024-03-25T18:32:51.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.977+0000 7ff587fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:51.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.977+0000 7ff587fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:51.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.977+0000 7ff587fff640 10 monclient(hunting): my global_id is 4593 2024-03-25T18:32:51.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.977+0000 7ff587fff640 1 -- 172.21.15.155:0/764475965 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5a0145710 con 0x7ff5a0140990 2024-03-25T18:32:51.980 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.978+0000 7ff587fff640 1 -- 172.21.15.155:0/764475965 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1024058655 0 0) 0x7ff58c004eb0 con 0x7ff5a00b00b0 2024-03-25T18:32:51.980 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.978+0000 7ff587fff640 1 -- 172.21.15.155:0/764475965 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff5a0135820 con 0x7ff5a00b00b0 2024-03-25T18:32:51.980 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.978+0000 7ff587fff640 1 -- 172.21.15.155:0/764475965 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7ff594001910 con 0x7ff5a00b00b0 2024-03-25T18:32:51.980 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.978+0000 7ff587fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:51.980 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.978+0000 7ff587fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:32:51.980 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.978+0000 7ff587fff640 10 monclient(hunting): dump: 2024-03-25T18:32:51.980 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:32:51.981 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:32:51.981 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:51.981 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:51.981 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:51.981 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:32:51.981 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:32:51.981 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:32:51.981 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:32:51.981 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:32:51.981 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.978+0000 7ff587fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:32:51.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.981+0000 7ff587fff640 1 -- 172.21.15.155:0/764475965 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3362028025 0 0) 0x7ff5a01459f0 con 0x7ff5a006ab80 2024-03-25T18:32:51.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.981+0000 7ff587fff640 10 monclient(hunting): _negotiate 2024-03-25T18:32:51.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.981+0000 7ff587fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:51.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.981+0000 7ff587fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:51.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.981+0000 7ff587fff640 10 monclient(hunting): my global_id is 4601 2024-03-25T18:32:51.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.981+0000 7ff587fff640 1 -- 172.21.15.155:0/764475965 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff58c004eb0 con 0x7ff5a006ab80 2024-03-25T18:32:51.987 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.986+0000 7ff587fff640 1 -- 172.21.15.155:0/764475965 <== mon.1 v1:172.21.15.167:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1447826708 0 0) 0x7ff5a0135820 con 0x7ff5a00b00b0 2024-03-25T18:32:51.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.986+0000 7ff587fff640 1 -- 172.21.15.155:0/764475965 >> v1:172.21.15.155:6790/0 conn(0x7ff5a006ab80 legacy=0x7ff5a01441f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:51.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.986+0000 7ff587fff640 1 -- 172.21.15.155:0/764475965 >> v1:172.21.15.155:6789/0 conn(0x7ff5a0140990 legacy=0x7ff5a0135110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:51.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.986+0000 7ff587fff640 10 monclient: _finish_hunting 0 2024-03-25T18:32:51.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.986+0000 7ff587fff640 1 monclient: found mon.b 2024-03-25T18:32:51.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.986+0000 7ff587fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:51.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.986+0000 7ff587fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:32:51.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.986+0000 7ff587fff640 1 -- 172.21.15.155:0/764475965 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5a0145cd0 con 0x7ff5a00b00b0 2024-03-25T18:32:51.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.986+0000 7ff587fff640 10 monclient: _finish_auth 0 2024-03-25T18:32:51.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.986+0000 7ff587fff640 10 monclient: _check_auth_tickets 2024-03-25T18:32:51.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.986+0000 7ff587fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:51.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.986+0000 7ff5a5ed3640 5 monclient: authenticate success, global_id 4588 2024-03-25T18:32:51.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.986+0000 7ff5a5ed3640 10 monclient: _renew_subs 2024-03-25T18:32:51.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.986+0000 7ff5a5ed3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:32:51.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.986+0000 7ff5a5ed3640 1 -- 172.21.15.155:0/764475965 --> v1:172.21.15.167:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5a0143cd0 con 0x7ff5a00b00b0 2024-03-25T18:32:51.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.986+0000 7ff5a5ed3640 10 client.4588.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:32:51.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.986+0000 7ff5a5ed3640 10 monclient: _renew_subs 2024-03-25T18:32:51.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.986+0000 7ff5a5ed3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:32:51.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.986+0000 7ff5a5ed3640 1 -- 172.21.15.155:0/764475965 --> v1:172.21.15.167:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5a0143720 con 0x7ff5a00b00b0 2024-03-25T18:32:51.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.986+0000 7ff5a5ed3640 1 librados: init done 2024-03-25T18:32:51.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.987+0000 7ff5a5ed3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:51.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.987+0000 7ff5a5ed3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:51.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.987+0000 7ff5a5ed3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:32:51.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:51.987+0000 7ff5a5ed3640 1 -- 172.21.15.155:0/764475965 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5a00627b0 con 0x7ff5a00b00b0 2024-03-25T18:32:52.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:52.265+0000 7ff587fff640 1 -- 172.21.15.155:0/764475965 <== mon.1 v1:172.21.15.167:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff5940047e0 con 0x7ff5a00b00b0 2024-03-25T18:32:52.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:52.265+0000 7ff587fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:52.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:52.265+0000 7ff587fff640 1 -- 172.21.15.155:0/764475965 <== mon.1 v1:172.21.15.167:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7ff594004bf0 con 0x7ff5a00b00b0 2024-03-25T18:32:52.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:52.265+0000 7ff587fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:52.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:52.265+0000 7ff587fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:32:52.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:52.265+0000 7ff587fff640 10 monclient: dump: 2024-03-25T18:32:52.267 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:32:52.267 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:32:52.267 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:52.267 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:52.267 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:52.267 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:32:52.268 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:32:52.268 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:32:52.268 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:32:52.268 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:32:52.268 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:52.265+0000 7ff59dffb640 4 set_mon_vals no callback set 2024-03-25T18:32:52.268 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:52.267+0000 7ff587fff640 1 -- 172.21.15.155:0/764475965 <== mon.1 v1:172.21.15.167:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7ff594004d50 con 0x7ff5a00b00b0 2024-03-25T18:32:52.269 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:52.268+0000 7ff59f7fe640 10 monclient: get_auth_request con 0x7ff58c06ddf0 auth_method 0 2024-03-25T18:32:52.269 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:52.268+0000 7ff587fff640 10 client.4588.objecter ms_handle_connect 0x7ff58c06ddf0 2024-03-25T18:32:52.271 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:52.270+0000 7ff587fff640 1 -- 172.21.15.155:0/764475965 <== mon.1 v1:172.21.15.167:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7ff5940548c0 con 0x7ff5a00b00b0 2024-03-25T18:32:52.271 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:52.270+0000 7ff587fff640 10 client.4588.objecter ms_dispatch 0x7ff5a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:32:52.271 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:52.270+0000 7ff587fff640 3 client.4588.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:32:52.272 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:52.270+0000 7ff587fff640 3 client.4588.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:32:52.272 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:52.270+0000 7ff587fff640 20 client.4588.objecter dump_active .. 0 homeless 2024-03-25T18:32:52.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:52.272+0000 7ff587fff640 1 -- 172.21.15.155:0/764475965 <== mon.1 v1:172.21.15.167:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff5940503d0 con 0x7ff5a00b00b0 2024-03-25T18:32:52.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:52.272+0000 7ff587fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:52.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:52.272+0000 7ff587fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:32:52.515 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:52.513+0000 7ff5a5ed3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:52.515 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:52.514+0000 7ff5a5ed3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:52.515 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:52.514+0000 7ff5a5ed3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:32:52.515 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:52.514+0000 7ff5a5ed3640 1 -- 172.21.15.155:0/764475965 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff5a0143e60 con 0x7ff5a00b00b0 2024-03-25T18:32:52.537 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:52.535+0000 7ff587fff640 1 -- 172.21.15.155:0/764475965 <== mon.1 v1:172.21.15.167:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7ff594052300 con 0x7ff5a00b00b0 2024-03-25T18:32:52.537 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:52.535+0000 7ff587fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:52.537 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:52.535+0000 7ff587fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:32:52.537 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:32:52.537 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:32:52.541 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:52.538+0000 7ff5a5ed3640 10 librados: watch_flush enter 2024-03-25T18:32:52.541 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:52.538+0000 7ff5a5ed3640 10 librados: watch_flush exit 2024-03-25T18:32:52.541 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:52.538+0000 7ff5a5ed3640 20 client.4588.objecter shutdown clearing up homeless session... 2024-03-25T18:32:52.541 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:52.538+0000 7ff5a5ed3640 10 client.4588.objecter successfully canceled tick 2024-03-25T18:32:52.541 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:52.538+0000 7ff5a5ed3640 1 -- 172.21.15.155:0/764475965 >> v1:172.21.15.167:6800/2919070503 conn(0x7ff58c06ddf0 legacy=0x7ff58c0702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:52.541 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:52.538+0000 7ff5a5ed3640 10 monclient: shutdown 2024-03-25T18:32:52.541 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:52.538+0000 7ff5a5ed3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:52.541 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:52.538+0000 7ff5a5ed3640 1 -- 172.21.15.155:0/764475965 >> v1:172.21.15.167:6789/0 conn(0x7ff5a00b00b0 legacy=0x7ff5a0134a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:52.541 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:52.539+0000 7ff5a5ed3640 1 -- 172.21.15.155:0/764475965 shutdown_connections 2024-03-25T18:32:52.541 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:52.539+0000 7ff5a5ed3640 1 -- 172.21.15.155:0/764475965 >> 172.21.15.155:0/764475965 conn(0x7ff5a00b4a80 msgr2=0x7ff5a00b4e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:52.541 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:52.539+0000 7ff5a5ed3640 1 -- 172.21.15.155:0/764475965 shutdown_connections 2024-03-25T18:32:52.541 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:52.539+0000 7ff5a5ed3640 1 -- 172.21.15.155:0/764475965 wait complete. 2024-03-25T18:32:52.542 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:52.539+0000 7ff5a5ed3640 1 librados: shutdown 2024-03-25T18:32:52.553 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:32:59.359 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:32:59.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.472+0000 7f2fd3e0e640 10 monclient: get_monmap_and_config 2024-03-25T18:32:59.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.472+0000 7f2fd3e0e640 10 monclient: build_initial_monmap 2024-03-25T18:32:59.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.472+0000 7f2fd3e0e640 10 monclient: monmap: 2024-03-25T18:32:59.474 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:32:59.474 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:32:59.474 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:32:59.473640+0000 2024-03-25T18:32:59.474 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:32:59.473640+0000 2024-03-25T18:32:59.474 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:59.475 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:32:59.475 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:32:59.475 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:32:59.475 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:32:59.475 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:32:59.477 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.473+0000 7f2fd3e0e640 1 Processor -- start 2024-03-25T18:32:59.477 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.473+0000 7f2fd3e0e640 1 -- start start 2024-03-25T18:32:59.477 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.473+0000 7f2fd3e0e640 10 monclient: init 2024-03-25T18:32:59.477 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.474+0000 7f2fd3e0e640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:59.477 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.474+0000 7f2fd3e0e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:32:59.477 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.474+0000 7f2fd3e0e640 10 monclient(hunting): picked mon.noname-a con 0x7f2fcc0b4d60 addr v1:172.21.15.155:6789/0 2024-03-25T18:32:59.477 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.474+0000 7f2fd3e0e640 10 monclient(hunting): picked mon.noname-c con 0x7f2fcc0b1730 addr v1:172.21.15.167:6789/0 2024-03-25T18:32:59.477 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.474+0000 7f2fd3e0e640 10 monclient(hunting): picked mon.noname-b con 0x7f2fcc14be90 addr v1:172.21.15.155:6790/0 2024-03-25T18:32:59.477 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.474+0000 7f2fd3e0e640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fcc0af940 con 0x7f2fcc0b4d60 2024-03-25T18:32:59.478 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.474+0000 7f2fd3e0e640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fcc0b0a70 con 0x7f2fcc0b1730 2024-03-25T18:32:59.478 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.474+0000 7f2fd3e0e640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fcc0b0d50 con 0x7f2fcc14be90 2024-03-25T18:32:59.478 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.474+0000 7f2fd3e0e640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:59.478 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.474+0000 7f2fd3e0e640 10 monclient(hunting): authenticate will time out at 8025.784371s 2024-03-25T18:32:59.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.479+0000 7f2fd13a9640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f2fcc0b1730 0x7f2fcc0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:47450/0 (socket says 172.21.15.155:47450) 2024-03-25T18:32:59.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.479+0000 7f2fd13a9640 1 -- 172.21.15.155:0/739129868 learned_addr learned my addr 172.21.15.155:0/739129868 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:32:59.495 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.493+0000 7f2fd0ba8640 1 -- 172.21.15.155:0/739129868 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1851670846 0 0) 0x7f2fcc0b0d50 con 0x7f2fcc14be90 2024-03-25T18:32:59.495 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.493+0000 7f2fd0ba8640 10 monclient(hunting): _negotiate 2024-03-25T18:32:59.495 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.493+0000 7f2fd0ba8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:59.495 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.493+0000 7f2fd0ba8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:59.495 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.493+0000 7f2fd0ba8640 10 monclient(hunting): my global_id is 4604 2024-03-25T18:32:59.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.493+0000 7f2fd0ba8640 1 -- 172.21.15.155:0/739129868 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fb8003520 con 0x7f2fcc14be90 2024-03-25T18:32:59.499 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.498+0000 7f2fd0ba8640 1 -- 172.21.15.155:0/739129868 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4040887960 0 0) 0x7f2fcc0b0a70 con 0x7f2fcc0b1730 2024-03-25T18:32:59.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.498+0000 7f2fd0ba8640 10 monclient(hunting): _negotiate 2024-03-25T18:32:59.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.498+0000 7f2fd0ba8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:59.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.498+0000 7f2fd0ba8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:59.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.498+0000 7f2fd0ba8640 10 monclient(hunting): my global_id is 4591 2024-03-25T18:32:59.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.498+0000 7f2fd0ba8640 1 -- 172.21.15.155:0/739129868 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fcc0b0d50 con 0x7f2fcc0b1730 2024-03-25T18:32:59.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.509+0000 7f2fd0ba8640 1 -- 172.21.15.155:0/739129868 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2782008767 0 0) 0x7f2fcc0af940 con 0x7f2fcc0b4d60 2024-03-25T18:32:59.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.509+0000 7f2fd0ba8640 10 monclient(hunting): _negotiate 2024-03-25T18:32:59.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.509+0000 7f2fd0ba8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:59.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.509+0000 7f2fd0ba8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:59.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.509+0000 7f2fd0ba8640 10 monclient(hunting): my global_id is 4596 2024-03-25T18:32:59.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.509+0000 7f2fd0ba8640 1 -- 172.21.15.155:0/739129868 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fcc0b0a70 con 0x7f2fcc0b4d60 2024-03-25T18:32:59.518 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.516+0000 7f2fd0ba8640 1 -- 172.21.15.155:0/739129868 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1479752424 0 0) 0x7f2fcc0b0d50 con 0x7f2fcc0b1730 2024-03-25T18:32:59.518 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.516+0000 7f2fd0ba8640 1 -- 172.21.15.155:0/739129868 >> v1:172.21.15.155:6790/0 conn(0x7f2fcc14be90 legacy=0x7f2fcc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:59.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.516+0000 7f2fd0ba8640 1 -- 172.21.15.155:0/739129868 >> v1:172.21.15.155:6789/0 conn(0x7f2fcc0b4d60 legacy=0x7f2fcc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:59.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.516+0000 7f2fd0ba8640 10 monclient: _finish_hunting 0 2024-03-25T18:32:59.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.516+0000 7f2fd0ba8640 1 monclient: found mon.noname-c 2024-03-25T18:32:59.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.516+0000 7f2fd0ba8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:59.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.516+0000 7f2fd0ba8640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.167:6789/0 2024-03-25T18:32:59.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.516+0000 7f2fd0ba8640 1 -- 172.21.15.155:0/739129868 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fcc0b1030 con 0x7f2fcc0b1730 2024-03-25T18:32:59.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.516+0000 7f2fd0ba8640 10 monclient: _finish_auth 0 2024-03-25T18:32:59.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.516+0000 7f2fd0ba8640 10 monclient: _check_auth_tickets 2024-03-25T18:32:59.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.516+0000 7f2fd0ba8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:59.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.516+0000 7f2fd0ba8640 1 -- 172.21.15.155:0/739129868 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f2fbc001670 con 0x7f2fcc0b1730 2024-03-25T18:32:59.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.516+0000 7f2fd0ba8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:59.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.516+0000 7f2fd0ba8640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:32:59.520 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.516+0000 7f2fd0ba8640 10 monclient: dump: 2024-03-25T18:32:59.520 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:32:59.520 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:32:59.520 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:59.520 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:59.520 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:59.520 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:32:59.520 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:32:59.520 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:32:59.520 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:32:59.520 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:32:59.520 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.516+0000 7f2fd3e0e640 5 monclient: authenticate success, global_id 4591 2024-03-25T18:32:59.520 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.516+0000 7f2fd3e0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:59.537 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.535+0000 7f2fd0ba8640 1 -- 172.21.15.155:0/739129868 <== mon.1 v1:172.21.15.167:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2fbc0017d0 con 0x7f2fcc0b1730 2024-03-25T18:32:59.537 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.536+0000 7f2fd0ba8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:59.537 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.536+0000 7f2fd0ba8640 1 -- 172.21.15.155:0/739129868 <== mon.1 v1:172.21.15.167:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f2fbc001be0 con 0x7f2fcc0b1730 2024-03-25T18:32:59.537 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.536+0000 7f2fd3e0e640 10 monclient: get_monmap_and_config success 2024-03-25T18:32:59.537 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.536+0000 7f2fd3e0e640 4 set_mon_vals no callback set 2024-03-25T18:32:59.537 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.536+0000 7f2fd0ba8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:59.537 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.536+0000 7f2fd0ba8640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:32:59.538 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.536+0000 7f2fd0ba8640 10 monclient: dump: 2024-03-25T18:32:59.538 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:32:59.538 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:32:59.538 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:59.538 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:59.538 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:59.538 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:32:59.538 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:32:59.538 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:32:59.538 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:32:59.538 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:32:59.540 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.536+0000 7f2fd3e0e640 10 monclient: shutdown 2024-03-25T18:32:59.540 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.536+0000 7f2fd3e0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:59.540 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.536+0000 7f2fd3e0e640 1 -- 172.21.15.155:0/739129868 >> v1:172.21.15.167:6789/0 conn(0x7f2fcc0b1730 legacy=0x7f2fcc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:59.540 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.536+0000 7f2fd3e0e640 1 -- 172.21.15.155:0/739129868 shutdown_connections 2024-03-25T18:32:59.540 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.536+0000 7f2fd3e0e640 1 -- 172.21.15.155:0/739129868 >> 172.21.15.155:0/739129868 conn(0x7f2fcc0617a0 msgr2=0x7f2fcc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:59.541 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.537+0000 7f2fd3e0e640 1 -- 172.21.15.155:0/739129868 shutdown_connections 2024-03-25T18:32:59.541 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.537+0000 7f2fd3e0e640 1 -- 172.21.15.155:0/739129868 wait complete. 2024-03-25T18:32:59.541 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.537+0000 7f2fd3e0e640 10 monclient: build_initial_monmap 2024-03-25T18:32:59.541 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.537+0000 7f2fd3e0e640 10 monclient: monmap: 2024-03-25T18:32:59.541 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:32:59.541 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:32:59.541 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:32:59.541 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:32:59.541 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:59.541 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:32:59.541 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:32:59.541 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:32:59.542 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:32:59.542 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:32:59.542 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.538+0000 7f2fd3e0e640 1 librados: starting msgr at 2024-03-25T18:32:59.542 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.538+0000 7f2fd3e0e640 1 librados: starting objecter 2024-03-25T18:32:59.542 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.538+0000 7f2fd3e0e640 1 Processor -- start 2024-03-25T18:32:59.542 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.538+0000 7f2fd3e0e640 1 -- start start 2024-03-25T18:32:59.542 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.538+0000 7f2fd3e0e640 1 librados: setting wanted keys 2024-03-25T18:32:59.542 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.538+0000 7f2fd3e0e640 1 librados: calling monclient init 2024-03-25T18:32:59.542 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.538+0000 7f2fd3e0e640 10 monclient: init 2024-03-25T18:32:59.542 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.538+0000 7f2fd3e0e640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:59.542 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.538+0000 7f2fd3e0e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:32:59.543 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.539+0000 7f2fd3e0e640 10 monclient(hunting): picked mon.noname-a con 0x7f2fcc0b1730 addr v1:172.21.15.155:6789/0 2024-03-25T18:32:59.543 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.539+0000 7f2fd3e0e640 10 monclient(hunting): picked mon.noname-c con 0x7f2fcc0b4d60 addr v1:172.21.15.155:6790/0 2024-03-25T18:32:59.543 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.539+0000 7f2fd3e0e640 10 monclient(hunting): picked mon.noname-b con 0x7f2fcc14be90 addr v1:172.21.15.167:6789/0 2024-03-25T18:32:59.543 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.539+0000 7f2fd3e0e640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fcc13ad90 con 0x7f2fcc0b1730 2024-03-25T18:32:59.543 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.539+0000 7f2fd3e0e640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fcc1dac30 con 0x7f2fcc14be90 2024-03-25T18:32:59.543 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.539+0000 7f2fd3e0e640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fcc1db860 con 0x7f2fcc0b4d60 2024-03-25T18:32:59.543 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.539+0000 7f2fd3e0e640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:59.543 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.539+0000 7f2fd3e0e640 10 monclient(hunting): authenticate will time out at 8025.848892s 2024-03-25T18:32:59.544 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.541+0000 7f2fd13a9640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f2fcc0b4d60 0x7f2fcc134d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:51928/0 (socket says 172.21.15.155:51928) 2024-03-25T18:32:59.544 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.541+0000 7f2fd13a9640 1 -- 172.21.15.155:0/3693741773 learned_addr learned my addr 172.21.15.155:0/3693741773 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:32:59.549 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.548+0000 7f2fc27fc640 10 client.?.objecter ms_handle_connect 0x7f2fcc0b4d60 2024-03-25T18:32:59.549 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.548+0000 7f2fc27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:59.561 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.560+0000 7f2fc27fc640 1 -- 172.21.15.155:0/3693741773 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1948243476 0 0) 0x7f2fcc1db860 con 0x7f2fcc0b4d60 2024-03-25T18:32:59.562 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.560+0000 7f2fc27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:32:59.562 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.560+0000 7f2fc27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:59.562 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.560+0000 7f2fc27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:59.562 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.560+0000 7f2fc27fc640 10 monclient(hunting): my global_id is 4607 2024-03-25T18:32:59.562 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.560+0000 7f2fc27fc640 1 -- 172.21.15.155:0/3693741773 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fb0004eb0 con 0x7f2fcc0b4d60 2024-03-25T18:32:59.564 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.563+0000 7f2fc27fc640 10 client.?.objecter ms_handle_connect 0x7f2fcc14be90 2024-03-25T18:32:59.564 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.563+0000 7f2fc27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:59.567 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.566+0000 7f2fc27fc640 10 client.?.objecter ms_handle_connect 0x7f2fcc0b1730 2024-03-25T18:32:59.567 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.566+0000 7f2fc27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:59.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.571+0000 7f2fc27fc640 1 -- 172.21.15.155:0/3693741773 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2978256825 0 0) 0x7f2fb0004eb0 con 0x7f2fcc0b4d60 2024-03-25T18:32:59.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.571+0000 7f2fc27fc640 1 -- 172.21.15.155:0/3693741773 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2fcc1db860 con 0x7f2fcc0b4d60 2024-03-25T18:32:59.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.571+0000 7f2fc27fc640 1 -- 172.21.15.155:0/3693741773 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f2fbc001c20 con 0x7f2fcc0b4d60 2024-03-25T18:32:59.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.571+0000 7f2fc27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:59.574 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.571+0000 7f2fc27fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:32:59.574 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.571+0000 7f2fc27fc640 10 monclient(hunting): dump: 2024-03-25T18:32:59.574 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:32:59.574 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:32:59.574 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:59.574 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:59.574 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:59.574 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:32:59.574 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:32:59.574 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:32:59.574 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:32:59.575 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:32:59.575 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.571+0000 7f2fc27fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:32:59.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.579+0000 7f2fc27fc640 1 -- 172.21.15.155:0/3693741773 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2134408926 0 0) 0x7f2fcc13ad90 con 0x7f2fcc0b1730 2024-03-25T18:32:59.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.579+0000 7f2fc27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:32:59.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.579+0000 7f2fc27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:59.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.579+0000 7f2fc27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:59.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.579+0000 7f2fc27fc640 10 monclient(hunting): my global_id is 4599 2024-03-25T18:32:59.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.579+0000 7f2fc27fc640 1 -- 172.21.15.155:0/3693741773 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fb0004eb0 con 0x7f2fcc0b1730 2024-03-25T18:32:59.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.581+0000 7f2fc27fc640 1 -- 172.21.15.155:0/3693741773 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2140017858 0 0) 0x7f2fcc1dac30 con 0x7f2fcc14be90 2024-03-25T18:32:59.583 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.581+0000 7f2fc27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:32:59.583 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.581+0000 7f2fc27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:59.583 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.581+0000 7f2fc27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:59.583 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.581+0000 7f2fc27fc640 10 monclient(hunting): my global_id is 4594 2024-03-25T18:32:59.583 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.581+0000 7f2fc27fc640 1 -- 172.21.15.155:0/3693741773 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fcc13ad90 con 0x7f2fcc14be90 2024-03-25T18:32:59.589 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.587+0000 7f2fc27fc640 1 -- 172.21.15.155:0/3693741773 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2401832355 0 0) 0x7f2fcc1db860 con 0x7f2fcc0b4d60 2024-03-25T18:32:59.589 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.587+0000 7f2fc27fc640 1 -- 172.21.15.155:0/3693741773 >> v1:172.21.15.167:6789/0 conn(0x7f2fcc14be90 legacy=0x7f2fcc1e2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:59.589 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.587+0000 7f2fc27fc640 1 -- 172.21.15.155:0/3693741773 >> v1:172.21.15.155:6789/0 conn(0x7f2fcc0b1730 legacy=0x7f2fcc1db150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:59.589 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.587+0000 7f2fc27fc640 10 monclient: _finish_hunting 0 2024-03-25T18:32:59.590 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.587+0000 7f2fc27fc640 1 monclient: found mon.c 2024-03-25T18:32:59.590 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.587+0000 7f2fc27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:59.590 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.587+0000 7f2fc27fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:32:59.590 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.587+0000 7f2fc27fc640 1 -- 172.21.15.155:0/3693741773 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fcc1db9f0 con 0x7f2fcc0b4d60 2024-03-25T18:32:59.590 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.587+0000 7f2fc27fc640 10 monclient: _finish_auth 0 2024-03-25T18:32:59.590 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.587+0000 7f2fc27fc640 10 monclient: _check_auth_tickets 2024-03-25T18:32:59.590 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.587+0000 7f2fc27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:59.590 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.587+0000 7f2fd3e0e640 5 monclient: authenticate success, global_id 4607 2024-03-25T18:32:59.590 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.587+0000 7f2fd3e0e640 10 monclient: _renew_subs 2024-03-25T18:32:59.590 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.587+0000 7f2fd3e0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:32:59.590 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.587+0000 7f2fd3e0e640 1 -- 172.21.15.155:0/3693741773 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fcc139a50 con 0x7f2fcc0b4d60 2024-03-25T18:32:59.590 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.587+0000 7f2fd3e0e640 10 client.4607.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:32:59.591 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.587+0000 7f2fd3e0e640 10 monclient: _renew_subs 2024-03-25T18:32:59.591 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.587+0000 7f2fd3e0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:32:59.591 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.587+0000 7f2fd3e0e640 1 -- 172.21.15.155:0/3693741773 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fcc139140 con 0x7f2fcc0b4d60 2024-03-25T18:32:59.591 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.587+0000 7f2fd3e0e640 1 librados: init done 2024-03-25T18:32:59.591 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.588+0000 7f2fd3e0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:59.591 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.588+0000 7f2fd3e0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:59.592 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.588+0000 7f2fd3e0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:32:59.592 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.588+0000 7f2fd3e0e640 1 -- 172.21.15.155:0/3693741773 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fcc0627b0 con 0x7f2fcc0b4d60 2024-03-25T18:32:59.899 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.897+0000 7f2fc27fc640 1 -- 172.21.15.155:0/3693741773 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2fbc004a90 con 0x7f2fcc0b4d60 2024-03-25T18:32:59.899 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.897+0000 7f2fc27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:59.899 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.897+0000 7f2fc27fc640 1 -- 172.21.15.155:0/3693741773 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f2fbc004e20 con 0x7f2fcc0b4d60 2024-03-25T18:32:59.899 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.897+0000 7f2fc27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:59.899 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.897+0000 7f2fc27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:32:59.899 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.897+0000 7f2fc27fc640 10 monclient: dump: 2024-03-25T18:32:59.899 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:32:59.899 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:32:59.899 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:59.899 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:32:59.899 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:59.899 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:32:59.900 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:32:59.900 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:32:59.900 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:32:59.900 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:32:59.900 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.897+0000 7f2fc3fff640 4 set_mon_vals no callback set 2024-03-25T18:32:59.900 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.898+0000 7f2fc27fc640 1 -- 172.21.15.155:0/3693741773 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7f2fbc005040 con 0x7f2fcc0b4d60 2024-03-25T18:32:59.900 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.898+0000 7f2fd1baa640 10 monclient: get_auth_request con 0x7f2fb006ddc0 auth_method 0 2024-03-25T18:32:59.901 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.898+0000 7f2fc27fc640 10 client.4607.objecter ms_handle_connect 0x7f2fb006ddc0 2024-03-25T18:32:59.904 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.903+0000 7f2fc27fc640 1 -- 172.21.15.155:0/3693741773 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7f2fbc0549e0 con 0x7f2fcc0b4d60 2024-03-25T18:32:59.905 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.903+0000 7f2fc27fc640 10 client.4607.objecter ms_dispatch 0x7f2fcc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:32:59.905 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.903+0000 7f2fc27fc640 3 client.4607.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:32:59.905 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.903+0000 7f2fc27fc640 3 client.4607.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:32:59.905 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.903+0000 7f2fc27fc640 20 client.4607.objecter dump_active .. 0 homeless 2024-03-25T18:32:59.905 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.903+0000 7f2fc27fc640 1 -- 172.21.15.155:0/3693741773 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2fbc0504f0 con 0x7f2fcc0b4d60 2024-03-25T18:32:59.905 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.903+0000 7f2fc27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:59.905 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:32:59.903+0000 7f2fc27fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:33:00.158 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:00.157+0000 7f2fd3e0e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:00.159 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:00.157+0000 7f2fd3e0e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:00.159 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:00.157+0000 7f2fd3e0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:33:00.159 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:00.157+0000 7f2fd3e0e640 1 -- 172.21.15.155:0/3693741773 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2fcc1dadc0 con 0x7f2fcc0b4d60 2024-03-25T18:33:00.183 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:00.182+0000 7f2fc27fc640 1 -- 172.21.15.155:0/3693741773 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7f2fbc052420 con 0x7f2fcc0b4d60 2024-03-25T18:33:00.183 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:00.182+0000 7f2fc27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:00.183 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:00.182+0000 7f2fc27fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:33:00.184 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:33:00.184 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:33:00.186 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:00.184+0000 7f2fd3e0e640 10 librados: watch_flush enter 2024-03-25T18:33:00.186 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:00.184+0000 7f2fd3e0e640 10 librados: watch_flush exit 2024-03-25T18:33:00.186 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:00.184+0000 7f2fd3e0e640 20 client.4607.objecter shutdown clearing up homeless session... 2024-03-25T18:33:00.186 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:00.184+0000 7f2fd3e0e640 10 client.4607.objecter successfully canceled tick 2024-03-25T18:33:00.186 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:00.184+0000 7f2fd3e0e640 1 -- 172.21.15.155:0/3693741773 >> v1:172.21.15.167:6800/2919070503 conn(0x7f2fb006ddc0 legacy=0x7f2fb0070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:00.186 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:00.184+0000 7f2fd3e0e640 10 monclient: shutdown 2024-03-25T18:33:00.186 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:00.184+0000 7f2fd3e0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:00.186 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:00.184+0000 7f2fd3e0e640 1 -- 172.21.15.155:0/3693741773 >> v1:172.21.15.155:6790/0 conn(0x7f2fcc0b4d60 legacy=0x7f2fcc134d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:00.187 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:00.184+0000 7f2fd3e0e640 1 -- 172.21.15.155:0/3693741773 shutdown_connections 2024-03-25T18:33:00.187 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:00.184+0000 7f2fd3e0e640 1 -- 172.21.15.155:0/3693741773 >> 172.21.15.155:0/3693741773 conn(0x7f2fcc0617a0 msgr2=0x7f2fcc13f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:00.187 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:00.184+0000 7f2fd3e0e640 1 -- 172.21.15.155:0/3693741773 shutdown_connections 2024-03-25T18:33:00.187 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:00.184+0000 7f2fd3e0e640 1 -- 172.21.15.155:0/3693741773 wait complete. 2024-03-25T18:33:00.187 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:00.185+0000 7f2fd3e0e640 1 librados: shutdown 2024-03-25T18:33:00.197 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:33:07.005 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:33:07.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.119+0000 7fe0b1055640 10 monclient: get_monmap_and_config 2024-03-25T18:33:07.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.119+0000 7fe0b1055640 10 monclient: build_initial_monmap 2024-03-25T18:33:07.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.119+0000 7fe0b1055640 10 monclient: monmap: 2024-03-25T18:33:07.121 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:33:07.121 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:33:07.121 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:33:07.120820+0000 2024-03-25T18:33:07.121 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:33:07.120820+0000 2024-03-25T18:33:07.122 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:07.122 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:33:07.122 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:33:07.122 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:33:07.122 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:33:07.122 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:33:07.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.120+0000 7fe0b1055640 1 Processor -- start 2024-03-25T18:33:07.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.120+0000 7fe0b1055640 1 -- start start 2024-03-25T18:33:07.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.120+0000 7fe0b1055640 10 monclient: init 2024-03-25T18:33:07.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.121+0000 7fe0b1055640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:07.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.121+0000 7fe0b1055640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:33:07.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.121+0000 7fe0b1055640 10 monclient(hunting): picked mon.noname-c con 0x7fe0ac0a98f0 addr v1:172.21.15.167:6789/0 2024-03-25T18:33:07.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.121+0000 7fe0b1055640 10 monclient(hunting): picked mon.noname-a con 0x7fe0ac13f710 addr v1:172.21.15.155:6789/0 2024-03-25T18:33:07.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.121+0000 7fe0b1055640 10 monclient(hunting): picked mon.noname-b con 0x7fe0ac1430c0 addr v1:172.21.15.155:6790/0 2024-03-25T18:33:07.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.121+0000 7fe0b1055640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0ac142520 con 0x7fe0ac13f710 2024-03-25T18:33:07.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.121+0000 7fe0b1055640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0ac0b28a0 con 0x7fe0ac0a98f0 2024-03-25T18:33:07.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.121+0000 7fe0b1055640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0ac0b2a30 con 0x7fe0ac1430c0 2024-03-25T18:33:07.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.121+0000 7fe0b1055640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:07.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.121+0000 7fe0b1055640 10 monclient(hunting): authenticate will time out at 8033.431073s 2024-03-25T18:33:07.126 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.125+0000 7fe0aad9d640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7fe0ac0a98f0 0x7fe0ac0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:41480/0 (socket says 172.21.15.155:41480) 2024-03-25T18:33:07.126 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.125+0000 7fe0aad9d640 1 -- 172.21.15.155:0/929836152 learned_addr learned my addr 172.21.15.155:0/929836152 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:33:07.176 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:08:40.780 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:07.183 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:08:40.787 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:07.207 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.205+0000 7fe0a9d9b640 1 -- 172.21.15.155:0/929836152 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2178668218 0 0) 0x7fe0ac142520 con 0x7fe0ac13f710 2024-03-25T18:33:07.207 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.205+0000 7fe0a9d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:33:07.207 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.205+0000 7fe0a9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:07.207 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.205+0000 7fe0a9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:07.207 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.205+0000 7fe0a9d9b640 10 monclient(hunting): my global_id is 4602 2024-03-25T18:33:07.207 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.205+0000 7fe0a9d9b640 1 -- 172.21.15.155:0/929836152 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe094003520 con 0x7fe0ac13f710 2024-03-25T18:33:07.211 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:08:40.816 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:07.231 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.230+0000 7fe0a9d9b640 1 -- 172.21.15.155:0/929836152 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1699875133 0 0) 0x7fe094003520 con 0x7fe0ac13f710 2024-03-25T18:33:07.231 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.230+0000 7fe0a9d9b640 1 -- 172.21.15.155:0/929836152 >> v1:172.21.15.155:6790/0 conn(0x7fe0ac1430c0 legacy=0x7fe0ac14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:07.231 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.230+0000 7fe0a9d9b640 1 -- 172.21.15.155:0/929836152 >> v1:172.21.15.167:6789/0 conn(0x7fe0ac0a98f0 legacy=0x7fe0ac0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:07.231 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.230+0000 7fe0a9d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:33:07.232 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.230+0000 7fe0a9d9b640 1 monclient: found mon.noname-a 2024-03-25T18:33:07.232 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.230+0000 7fe0a9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:07.232 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.230+0000 7fe0a9d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-25T18:33:07.232 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.230+0000 7fe0a9d9b640 1 -- 172.21.15.155:0/929836152 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0ac0b44c0 con 0x7fe0ac13f710 2024-03-25T18:33:07.232 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.230+0000 7fe0a9d9b640 10 monclient: _finish_auth 0 2024-03-25T18:33:07.232 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.230+0000 7fe0a9d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:33:07.232 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.230+0000 7fe0a9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:07.232 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.230+0000 7fe0a9d9b640 1 -- 172.21.15.155:0/929836152 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fe0a0001690 con 0x7fe0ac13f710 2024-03-25T18:33:07.232 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.230+0000 7fe0a9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:07.232 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.230+0000 7fe0a9d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:33:07.232 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.230+0000 7fe0a9d9b640 10 monclient: dump: 2024-03-25T18:33:07.232 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:33:07.232 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:33:07.233 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:07.233 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:07.233 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:07.233 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:33:07.233 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:33:07.233 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:33:07.233 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:33:07.233 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:33:07.233 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.230+0000 7fe0b1055640 5 monclient: authenticate success, global_id 4602 2024-03-25T18:33:07.233 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.230+0000 7fe0b1055640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:07.234 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:08:40.811 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:07.238 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:08:40.817 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:07.249 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:08:40.853 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:07.255 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.254+0000 7fe0a9d9b640 1 -- 172.21.15.155:0/929836152 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0a00017f0 con 0x7fe0ac13f710 2024-03-25T18:33:07.255 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.254+0000 7fe0a9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:07.255 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.254+0000 7fe0a9d9b640 1 -- 172.21.15.155:0/929836152 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fe0a0001c00 con 0x7fe0ac13f710 2024-03-25T18:33:07.255 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.254+0000 7fe0a9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:07.255 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.254+0000 7fe0a9d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:33:07.256 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.254+0000 7fe0a9d9b640 10 monclient: dump: 2024-03-25T18:33:07.256 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:33:07.256 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:33:07.256 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:07.256 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:07.256 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:07.256 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:33:07.256 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:33:07.256 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:33:07.256 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:33:07.256 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:33:07.256 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.254+0000 7fe0b1055640 10 monclient: get_monmap_and_config success 2024-03-25T18:33:07.256 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.254+0000 7fe0b1055640 4 set_mon_vals no callback set 2024-03-25T18:33:07.258 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.255+0000 7fe0b1055640 10 monclient: shutdown 2024-03-25T18:33:07.258 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.255+0000 7fe0b1055640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:07.258 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.255+0000 7fe0b1055640 1 -- 172.21.15.155:0/929836152 >> v1:172.21.15.155:6789/0 conn(0x7fe0ac13f710 legacy=0x7fe0ac1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:07.258 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.255+0000 7fe0b1055640 1 -- 172.21.15.155:0/929836152 shutdown_connections 2024-03-25T18:33:07.258 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.255+0000 7fe0b1055640 1 -- 172.21.15.155:0/929836152 >> 172.21.15.155:0/929836152 conn(0x7fe0ac0b7ea0 msgr2=0x7fe0ac0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:07.258 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.255+0000 7fe0b1055640 1 -- 172.21.15.155:0/929836152 shutdown_connections 2024-03-25T18:33:07.258 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.255+0000 7fe0b1055640 1 -- 172.21.15.155:0/929836152 wait complete. 2024-03-25T18:33:07.258 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.256+0000 7fe0b1055640 10 monclient: build_initial_monmap 2024-03-25T18:33:07.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.256+0000 7fe0b1055640 10 monclient: monmap: 2024-03-25T18:33:07.259 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:33:07.259 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:33:07.259 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:33:07.259 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:33:07.259 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:07.259 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:33:07.259 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:33:07.259 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:33:07.259 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:33:07.259 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:33:07.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.256+0000 7fe0b1055640 1 librados: starting msgr at 2024-03-25T18:33:07.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.256+0000 7fe0b1055640 1 librados: starting objecter 2024-03-25T18:33:07.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.256+0000 7fe0b1055640 1 Processor -- start 2024-03-25T18:33:07.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.256+0000 7fe0b1055640 1 -- start start 2024-03-25T18:33:07.260 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.256+0000 7fe0b1055640 1 librados: setting wanted keys 2024-03-25T18:33:07.260 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.256+0000 7fe0b1055640 1 librados: calling monclient init 2024-03-25T18:33:07.260 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.256+0000 7fe0b1055640 10 monclient: init 2024-03-25T18:33:07.260 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.257+0000 7fe0b1055640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:07.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.257+0000 7fe0b1055640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:33:07.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.257+0000 7fe0b1055640 10 monclient(hunting): picked mon.noname-a con 0x7fe0ac0a98f0 addr v1:172.21.15.155:6789/0 2024-03-25T18:33:07.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.257+0000 7fe0b1055640 10 monclient(hunting): picked mon.noname-b con 0x7fe0ac13f710 addr v1:172.21.15.167:6789/0 2024-03-25T18:33:07.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.257+0000 7fe0b1055640 10 monclient(hunting): picked mon.noname-c con 0x7fe0ac1430c0 addr v1:172.21.15.155:6790/0 2024-03-25T18:33:07.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.257+0000 7fe0b1055640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0ac0abac0 con 0x7fe0ac0a98f0 2024-03-25T18:33:07.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.257+0000 7fe0b1055640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0ac0abda0 con 0x7fe0ac13f710 2024-03-25T18:33:07.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.257+0000 7fe0b1055640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0ac1380e0 con 0x7fe0ac1430c0 2024-03-25T18:33:07.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.257+0000 7fe0b1055640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:07.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.257+0000 7fe0b1055640 10 monclient(hunting): authenticate will time out at 8033.567240s 2024-03-25T18:33:07.261 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:08:40.863 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:07.262 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:08:40.840 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:07.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.278+0000 7fe0ab59e640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fe0ac1430c0 0x7fe0ac1e26b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:50518/0 (socket says 172.21.15.155:50518) 2024-03-25T18:33:07.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.278+0000 7fe0ab59e640 1 -- 172.21.15.155:0/3049645804 learned_addr learned my addr 172.21.15.155:0/3049645804 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:33:07.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.280+0000 7fe0937fe640 10 client.?.objecter ms_handle_connect 0x7fe0ac1430c0 2024-03-25T18:33:07.282 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.280+0000 7fe0937fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:07.282 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:08:40.886 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:07.333 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.332+0000 7fe0937fe640 10 client.?.objecter ms_handle_connect 0x7fe0ac13f710 2024-03-25T18:33:07.334 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.332+0000 7fe0937fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:07.362 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.361+0000 7fe0937fe640 10 client.?.objecter ms_handle_connect 0x7fe0ac0a98f0 2024-03-25T18:33:07.362 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.361+0000 7fe0937fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:07.395 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.393+0000 7fe0937fe640 1 -- 172.21.15.155:0/3049645804 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 531683500 0 0) 0x7fe0ac0abac0 con 0x7fe0ac0a98f0 2024-03-25T18:33:07.395 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.393+0000 7fe0937fe640 10 monclient(hunting): _negotiate 2024-03-25T18:33:07.395 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.393+0000 7fe0937fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:07.395 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.393+0000 7fe0937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:07.395 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.393+0000 7fe0937fe640 10 monclient(hunting): my global_id is 4605 2024-03-25T18:33:07.395 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.393+0000 7fe0937fe640 1 -- 172.21.15.155:0/3049645804 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe098004eb0 con 0x7fe0ac0a98f0 2024-03-25T18:33:07.425 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.424+0000 7fe0937fe640 1 -- 172.21.15.155:0/3049645804 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3453425778 0 0) 0x7fe0ac0abda0 con 0x7fe0ac13f710 2024-03-25T18:33:07.426 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.424+0000 7fe0937fe640 10 monclient(hunting): _negotiate 2024-03-25T18:33:07.426 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.424+0000 7fe0937fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:07.426 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.424+0000 7fe0937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:07.426 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.424+0000 7fe0937fe640 10 monclient(hunting): my global_id is 4600 2024-03-25T18:33:07.426 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.424+0000 7fe0937fe640 1 -- 172.21.15.155:0/3049645804 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0ac0abac0 con 0x7fe0ac13f710 2024-03-25T18:33:07.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.430+0000 7fe0937fe640 1 -- 172.21.15.155:0/3049645804 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2993392930 0 0) 0x7fe0ac1380e0 con 0x7fe0ac1430c0 2024-03-25T18:33:07.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.430+0000 7fe0937fe640 10 monclient(hunting): _negotiate 2024-03-25T18:33:07.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.430+0000 7fe0937fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:07.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.430+0000 7fe0937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:07.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.430+0000 7fe0937fe640 10 monclient(hunting): my global_id is 4613 2024-03-25T18:33:07.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.430+0000 7fe0937fe640 1 -- 172.21.15.155:0/3049645804 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0ac0abda0 con 0x7fe0ac1430c0 2024-03-25T18:33:07.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.435+0000 7fe0937fe640 1 -- 172.21.15.155:0/3049645804 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2022577312 0 0) 0x7fe098004eb0 con 0x7fe0ac0a98f0 2024-03-25T18:33:07.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.435+0000 7fe0937fe640 1 -- 172.21.15.155:0/3049645804 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0ac1380e0 con 0x7fe0ac0a98f0 2024-03-25T18:33:07.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.435+0000 7fe0937fe640 1 -- 172.21.15.155:0/3049645804 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fe09c0018a0 con 0x7fe0ac0a98f0 2024-03-25T18:33:07.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.435+0000 7fe0937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:07.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.435+0000 7fe0937fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:33:07.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.435+0000 7fe0937fe640 10 monclient(hunting): dump: 2024-03-25T18:33:07.437 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:33:07.437 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:33:07.437 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:07.437 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:07.437 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:07.437 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:33:07.437 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:33:07.437 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:33:07.437 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:33:07.437 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:33:07.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.435+0000 7fe0937fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:33:07.447 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.445+0000 7fe0937fe640 1 -- 172.21.15.155:0/3049645804 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1025520917 0 0) 0x7fe0ac0abac0 con 0x7fe0ac13f710 2024-03-25T18:33:07.447 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.446+0000 7fe0937fe640 1 -- 172.21.15.155:0/3049645804 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe098004eb0 con 0x7fe0ac13f710 2024-03-25T18:33:07.447 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.446+0000 7fe0937fe640 1 -- 172.21.15.155:0/3049645804 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fe0a0001b80 con 0x7fe0ac13f710 2024-03-25T18:33:07.447 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.446+0000 7fe0937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:07.447 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.446+0000 7fe0937fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:33:07.447 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.446+0000 7fe0937fe640 10 monclient(hunting): dump: 2024-03-25T18:33:07.447 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:33:07.447 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:33:07.448 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:07.448 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:07.448 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:07.448 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:33:07.448 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:33:07.448 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:33:07.448 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:33:07.448 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:33:07.470 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.468+0000 7fe0937fe640 1 -- 172.21.15.155:0/3049645804 <== mon.1 v1:172.21.15.167:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3272287865 0 0) 0x7fe098004eb0 con 0x7fe0ac13f710 2024-03-25T18:33:07.470 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.468+0000 7fe0937fe640 1 -- 172.21.15.155:0/3049645804 >> v1:172.21.15.155:6790/0 conn(0x7fe0ac1430c0 legacy=0x7fe0ac1e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:07.470 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.468+0000 7fe0937fe640 1 -- 172.21.15.155:0/3049645804 >> v1:172.21.15.155:6789/0 conn(0x7fe0ac0a98f0 legacy=0x7fe0ac0ac580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:07.470 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.468+0000 7fe0937fe640 10 monclient: _finish_hunting 0 2024-03-25T18:33:07.471 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.468+0000 7fe0937fe640 1 monclient: found mon.b 2024-03-25T18:33:07.471 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.468+0000 7fe0937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:07.471 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.468+0000 7fe0937fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:33:07.471 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.468+0000 7fe0937fe640 1 -- 172.21.15.155:0/3049645804 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0ac137d90 con 0x7fe0ac13f710 2024-03-25T18:33:07.471 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.468+0000 7fe0937fe640 10 monclient: _finish_auth 0 2024-03-25T18:33:07.471 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.468+0000 7fe0937fe640 10 monclient: _check_auth_tickets 2024-03-25T18:33:07.471 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.468+0000 7fe0937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:07.471 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.468+0000 7fe0b1055640 5 monclient: authenticate success, global_id 4600 2024-03-25T18:33:07.471 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.468+0000 7fe0b1055640 10 monclient: _renew_subs 2024-03-25T18:33:07.471 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.468+0000 7fe0b1055640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:33:07.471 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.468+0000 7fe0b1055640 1 -- 172.21.15.155:0/3049645804 --> v1:172.21.15.167:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0ac139240 con 0x7fe0ac13f710 2024-03-25T18:33:07.471 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.469+0000 7fe0b1055640 10 client.4600.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:33:07.471 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.469+0000 7fe0b1055640 10 monclient: _renew_subs 2024-03-25T18:33:07.471 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.469+0000 7fe0b1055640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:33:07.471 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.469+0000 7fe0b1055640 1 -- 172.21.15.155:0/3049645804 --> v1:172.21.15.167:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0ac136830 con 0x7fe0ac13f710 2024-03-25T18:33:07.471 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.469+0000 7fe0b1055640 1 librados: init done 2024-03-25T18:33:07.472 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.470+0000 7fe0b1055640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:07.472 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.470+0000 7fe0b1055640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:07.472 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.470+0000 7fe0b1055640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:33:07.472 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.470+0000 7fe0b1055640 1 -- 172.21.15.155:0/3049645804 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0ac0627b0 con 0x7fe0ac13f710 2024-03-25T18:33:07.754 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.753+0000 7fe0937fe640 1 -- 172.21.15.155:0/3049645804 <== mon.1 v1:172.21.15.167:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0a00038a0 con 0x7fe0ac13f710 2024-03-25T18:33:07.754 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.753+0000 7fe0937fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:07.754 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.753+0000 7fe0937fe640 1 -- 172.21.15.155:0/3049645804 <== mon.1 v1:172.21.15.167:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fe0a0001250 con 0x7fe0ac13f710 2024-03-25T18:33:07.754 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.753+0000 7fe0937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:07.755 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.753+0000 7fe0937fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:33:07.755 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.753+0000 7fe0937fe640 10 monclient: dump: 2024-03-25T18:33:07.755 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:33:07.755 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:33:07.755 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:07.755 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:07.755 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:07.755 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:33:07.755 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:33:07.755 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:33:07.755 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:33:07.755 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:33:07.755 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.753+0000 7fe0a959a640 4 set_mon_vals no callback set 2024-03-25T18:33:07.756 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.754+0000 7fe0937fe640 1 -- 172.21.15.155:0/3049645804 <== mon.1 v1:172.21.15.167:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7fe0a0019ae0 con 0x7fe0ac13f710 2024-03-25T18:33:07.756 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.754+0000 7fe0aad9d640 10 monclient: get_auth_request con 0x7fe09806dfb0 auth_method 0 2024-03-25T18:33:07.756 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.754+0000 7fe0937fe640 10 client.4600.objecter ms_handle_connect 0x7fe09806dfb0 2024-03-25T18:33:07.760 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.758+0000 7fe0937fe640 1 -- 172.21.15.155:0/3049645804 <== mon.1 v1:172.21.15.167:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7fe0a00536e0 con 0x7fe0ac13f710 2024-03-25T18:33:07.760 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.758+0000 7fe0937fe640 10 client.4600.objecter ms_dispatch 0x7fe0ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:33:07.760 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.758+0000 7fe0937fe640 3 client.4600.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:33:07.760 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.758+0000 7fe0937fe640 3 client.4600.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:33:07.760 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.758+0000 7fe0937fe640 20 client.4600.objecter dump_active .. 0 homeless 2024-03-25T18:33:07.760 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.759+0000 7fe0937fe640 1 -- 172.21.15.155:0/3049645804 <== mon.1 v1:172.21.15.167:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe0a004f020 con 0x7fe0ac13f710 2024-03-25T18:33:07.760 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.759+0000 7fe0937fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:07.760 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.759+0000 7fe0937fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:33:08.001 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:07.999+0000 7fe0b1055640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:08.001 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:08.000+0000 7fe0b1055640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:08.001 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:08.000+0000 7fe0b1055640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:33:08.001 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:08.000+0000 7fe0b1055640 1 -- 172.21.15.155:0/3049645804 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe0ac136370 con 0x7fe0ac13f710 2024-03-25T18:33:08.044 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:08.043+0000 7fe0937fe640 1 -- 172.21.15.155:0/3049645804 <== mon.1 v1:172.21.15.167:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7fe0ac136370 con 0x7fe0ac13f710 2024-03-25T18:33:08.044 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:08.043+0000 7fe0937fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:08.044 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:08.043+0000 7fe0937fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:33:08.044 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:33:08.044 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:33:08.047 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:08.045+0000 7fe0b1055640 10 librados: watch_flush enter 2024-03-25T18:33:08.047 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:08.045+0000 7fe0b1055640 10 librados: watch_flush exit 2024-03-25T18:33:08.047 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:08.045+0000 7fe0b1055640 20 client.4600.objecter shutdown clearing up homeless session... 2024-03-25T18:33:08.047 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:08.045+0000 7fe0b1055640 10 client.4600.objecter successfully canceled tick 2024-03-25T18:33:08.047 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:08.045+0000 7fe0b1055640 1 -- 172.21.15.155:0/3049645804 >> v1:172.21.15.167:6800/2919070503 conn(0x7fe09806dfb0 legacy=0x7fe098070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:08.047 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:08.045+0000 7fe0b1055640 10 monclient: shutdown 2024-03-25T18:33:08.047 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:08.045+0000 7fe0b1055640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:08.047 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:08.045+0000 7fe0b1055640 1 -- 172.21.15.155:0/3049645804 >> v1:172.21.15.167:6789/0 conn(0x7fe0ac13f710 legacy=0x7fe0ac138b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:08.047 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:08.045+0000 7fe0b1055640 1 -- 172.21.15.155:0/3049645804 shutdown_connections 2024-03-25T18:33:08.047 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:08.045+0000 7fe0b1055640 1 -- 172.21.15.155:0/3049645804 >> 172.21.15.155:0/3049645804 conn(0x7fe0ac0b7ea0 msgr2=0x7fe0ac074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:08.047 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:08.045+0000 7fe0b1055640 1 -- 172.21.15.155:0/3049645804 shutdown_connections 2024-03-25T18:33:08.047 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:08.045+0000 7fe0b1055640 1 -- 172.21.15.155:0/3049645804 wait complete. 2024-03-25T18:33:08.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:08.045+0000 7fe0b1055640 1 librados: shutdown 2024-03-25T18:33:08.058 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:33:14.866 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:33:14.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:14.980+0000 7f68ae38f640 10 monclient: get_monmap_and_config 2024-03-25T18:33:14.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:14.980+0000 7f68ae38f640 10 monclient: build_initial_monmap 2024-03-25T18:33:14.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:14.980+0000 7f68ae38f640 10 monclient: monmap: 2024-03-25T18:33:14.982 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:33:14.982 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:33:14.982 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:33:14.981715+0000 2024-03-25T18:33:14.982 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:33:14.981715+0000 2024-03-25T18:33:14.982 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:14.982 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:33:14.983 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:33:14.983 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:33:14.983 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:33:14.983 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:33:14.984 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:14.981+0000 7f68ae38f640 1 Processor -- start 2024-03-25T18:33:14.984 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:14.981+0000 7f68ae38f640 1 -- start start 2024-03-25T18:33:14.984 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:14.981+0000 7f68ae38f640 10 monclient: init 2024-03-25T18:33:14.984 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:14.982+0000 7f68ae38f640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:14.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:14.982+0000 7f68ae38f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:33:14.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:14.982+0000 7f68ae38f640 10 monclient(hunting): picked mon.noname-b con 0x7f68a80a98f0 addr v1:172.21.15.155:6790/0 2024-03-25T18:33:14.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:14.982+0000 7f68ae38f640 10 monclient(hunting): picked mon.noname-a con 0x7f68a813f710 addr v1:172.21.15.155:6789/0 2024-03-25T18:33:14.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:14.982+0000 7f68ae38f640 10 monclient(hunting): picked mon.noname-c con 0x7f68a81430c0 addr v1:172.21.15.167:6789/0 2024-03-25T18:33:14.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:14.982+0000 7f68ae38f640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68a8142520 con 0x7f68a813f710 2024-03-25T18:33:14.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:14.982+0000 7f68ae38f640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68a80b28a0 con 0x7f68a81430c0 2024-03-25T18:33:14.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:14.983+0000 7f68ae38f640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68a80b2a30 con 0x7f68a80a98f0 2024-03-25T18:33:14.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:14.983+0000 7f68ae38f640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:14.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:14.983+0000 7f68ae38f640 10 monclient(hunting): authenticate will time out at 8041.292887s 2024-03-25T18:33:14.991 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:14.990+0000 7f68a77fe640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f68a813f710 0x7f68a81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:52382/0 (socket says 172.21.15.155:52382) 2024-03-25T18:33:14.991 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:14.990+0000 7f68a77fe640 1 -- 172.21.15.155:0/1042534985 learned_addr learned my addr 172.21.15.155:0/1042534985 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:33:15.009 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.008+0000 7f68a6ffd640 1 -- 172.21.15.155:0/1042534985 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2779619667 0 0) 0x7f68a80b28a0 con 0x7f68a81430c0 2024-03-25T18:33:15.009 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.008+0000 7f68a6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:15.009 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.008+0000 7f68a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:15.009 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.008+0000 7f68a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:15.009 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.008+0000 7f68a6ffd640 10 monclient(hunting): my global_id is 4603 2024-03-25T18:33:15.010 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.008+0000 7f68a6ffd640 1 -- 172.21.15.155:0/1042534985 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6890003520 con 0x7f68a81430c0 2024-03-25T18:33:15.010 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.009+0000 7f68a6ffd640 1 -- 172.21.15.155:0/1042534985 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4157577368 0 0) 0x7f68a80b2a30 con 0x7f68a80a98f0 2024-03-25T18:33:15.011 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.009+0000 7f68a6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:15.011 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.009+0000 7f68a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:15.011 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.009+0000 7f68a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:15.011 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.009+0000 7f68a6ffd640 10 monclient(hunting): my global_id is 4616 2024-03-25T18:33:15.011 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.009+0000 7f68a6ffd640 1 -- 172.21.15.155:0/1042534985 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68a80b28a0 con 0x7f68a80a98f0 2024-03-25T18:33:15.016 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.014+0000 7f68a6ffd640 1 -- 172.21.15.155:0/1042534985 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 338956652 0 0) 0x7f68a8142520 con 0x7f68a813f710 2024-03-25T18:33:15.016 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.014+0000 7f68a6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:15.016 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.014+0000 7f68a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:15.016 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.014+0000 7f68a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:15.016 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.014+0000 7f68a6ffd640 10 monclient(hunting): my global_id is 4608 2024-03-25T18:33:15.016 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.014+0000 7f68a6ffd640 1 -- 172.21.15.155:0/1042534985 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68a80b2a30 con 0x7f68a813f710 2024-03-25T18:33:15.025 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.023+0000 7f68a6ffd640 1 -- 172.21.15.155:0/1042534985 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3822379410 0 0) 0x7f6890003520 con 0x7f68a81430c0 2024-03-25T18:33:15.025 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.023+0000 7f68a6ffd640 1 -- 172.21.15.155:0/1042534985 >> v1:172.21.15.155:6790/0 conn(0x7f68a80a98f0 legacy=0x7f68a80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:15.025 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.023+0000 7f68a6ffd640 1 -- 172.21.15.155:0/1042534985 >> v1:172.21.15.155:6789/0 conn(0x7f68a813f710 legacy=0x7f68a81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:15.025 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.023+0000 7f68a6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:33:15.025 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.023+0000 7f68a6ffd640 1 monclient: found mon.noname-c 2024-03-25T18:33:15.025 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.023+0000 7f68a6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:15.025 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.023+0000 7f68a6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.167:6789/0 2024-03-25T18:33:15.025 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.023+0000 7f68a6ffd640 1 -- 172.21.15.155:0/1042534985 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68a80b44c0 con 0x7f68a81430c0 2024-03-25T18:33:15.025 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.023+0000 7f68a6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:33:15.025 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.023+0000 7f68a6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:33:15.026 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.023+0000 7f68a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:15.026 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.023+0000 7f68a6ffd640 1 -- 172.21.15.155:0/1042534985 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f68a0001a10 con 0x7f68a81430c0 2024-03-25T18:33:15.026 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.023+0000 7f68a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:15.026 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.023+0000 7f68a6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:33:15.026 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.023+0000 7f68a6ffd640 10 monclient: dump: 2024-03-25T18:33:15.026 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:33:15.026 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:33:15.026 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:15.026 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:15.026 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:15.026 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:33:15.026 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:33:15.026 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:33:15.026 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:33:15.027 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:33:15.027 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.023+0000 7f68ae38f640 5 monclient: authenticate success, global_id 4603 2024-03-25T18:33:15.027 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.023+0000 7f68ae38f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:15.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.030+0000 7f68a6ffd640 1 -- 172.21.15.155:0/1042534985 <== mon.1 v1:172.21.15.167:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f68a0001b70 con 0x7f68a81430c0 2024-03-25T18:33:15.033 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.030+0000 7f68a6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:15.033 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.030+0000 7f68a6ffd640 1 -- 172.21.15.155:0/1042534985 <== mon.1 v1:172.21.15.167:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f68a0001f80 con 0x7f68a81430c0 2024-03-25T18:33:15.033 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.030+0000 7f68a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:15.033 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.030+0000 7f68a6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:33:15.033 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.030+0000 7f68a6ffd640 10 monclient: dump: 2024-03-25T18:33:15.033 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:33:15.033 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:33:15.033 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:15.033 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:15.033 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:15.033 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:33:15.033 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:33:15.033 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:33:15.034 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:33:15.034 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:33:15.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.030+0000 7f68ae38f640 10 monclient: get_monmap_and_config success 2024-03-25T18:33:15.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.030+0000 7f68ae38f640 4 set_mon_vals no callback set 2024-03-25T18:33:15.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.031+0000 7f68ae38f640 10 monclient: shutdown 2024-03-25T18:33:15.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.031+0000 7f68ae38f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:15.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.031+0000 7f68ae38f640 1 -- 172.21.15.155:0/1042534985 >> v1:172.21.15.167:6789/0 conn(0x7f68a81430c0 legacy=0x7f68a814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:15.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.031+0000 7f68ae38f640 1 -- 172.21.15.155:0/1042534985 shutdown_connections 2024-03-25T18:33:15.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.031+0000 7f68ae38f640 1 -- 172.21.15.155:0/1042534985 >> 172.21.15.155:0/1042534985 conn(0x7f68a80b7ea0 msgr2=0x7f68a80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:15.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.031+0000 7f68ae38f640 1 -- 172.21.15.155:0/1042534985 shutdown_connections 2024-03-25T18:33:15.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.031+0000 7f68ae38f640 1 -- 172.21.15.155:0/1042534985 wait complete. 2024-03-25T18:33:15.036 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.032+0000 7f68ae38f640 10 monclient: build_initial_monmap 2024-03-25T18:33:15.036 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.032+0000 7f68ae38f640 10 monclient: monmap: 2024-03-25T18:33:15.036 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:33:15.036 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:33:15.036 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:33:15.036 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:33:15.036 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:15.036 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:33:15.036 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:33:15.036 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:33:15.037 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:33:15.037 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:33:15.037 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.032+0000 7f68ae38f640 1 librados: starting msgr at 2024-03-25T18:33:15.037 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.032+0000 7f68ae38f640 1 librados: starting objecter 2024-03-25T18:33:15.037 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.033+0000 7f68ae38f640 1 Processor -- start 2024-03-25T18:33:15.037 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.033+0000 7f68ae38f640 1 -- start start 2024-03-25T18:33:15.037 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.033+0000 7f68ae38f640 1 librados: setting wanted keys 2024-03-25T18:33:15.037 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.033+0000 7f68ae38f640 1 librados: calling monclient init 2024-03-25T18:33:15.037 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.033+0000 7f68ae38f640 10 monclient: init 2024-03-25T18:33:15.037 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.034+0000 7f68ae38f640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:15.037 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.034+0000 7f68ae38f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:33:15.037 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.034+0000 7f68ae38f640 10 monclient(hunting): picked mon.noname-a con 0x7f68a80a98f0 addr v1:172.21.15.155:6789/0 2024-03-25T18:33:15.037 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.034+0000 7f68ae38f640 10 monclient(hunting): picked mon.noname-c con 0x7f68a813f710 addr v1:172.21.15.155:6790/0 2024-03-25T18:33:15.037 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.034+0000 7f68ae38f640 10 monclient(hunting): picked mon.noname-b con 0x7f68a81430c0 addr v1:172.21.15.167:6789/0 2024-03-25T18:33:15.038 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.034+0000 7f68ae38f640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68a814e580 con 0x7f68a80a98f0 2024-03-25T18:33:15.038 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.034+0000 7f68ae38f640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68a814e7d0 con 0x7f68a81430c0 2024-03-25T18:33:15.038 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.034+0000 7f68ae38f640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68a8139990 con 0x7f68a813f710 2024-03-25T18:33:15.038 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.034+0000 7f68ae38f640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:15.038 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.034+0000 7f68ae38f640 10 monclient(hunting): authenticate will time out at 8041.343930s 2024-03-25T18:33:15.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.047+0000 7f68a77fe640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f68a813f710 0x7f68a813a6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:55470/0 (socket says 172.21.15.155:55470) 2024-03-25T18:33:15.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.047+0000 7f68a77fe640 1 -- 172.21.15.155:0/1518028703 learned_addr learned my addr 172.21.15.155:0/1518028703 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:33:15.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.049+0000 7f68a4ff9640 10 client.?.objecter ms_handle_connect 0x7f68a813f710 2024-03-25T18:33:15.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.049+0000 7f68a4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:15.055 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.054+0000 7f68a4ff9640 10 client.?.objecter ms_handle_connect 0x7f68a80a98f0 2024-03-25T18:33:15.056 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.054+0000 7f68a4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:15.058 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.057+0000 7f68a4ff9640 10 client.?.objecter ms_handle_connect 0x7f68a81430c0 2024-03-25T18:33:15.058 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.057+0000 7f68a4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:15.068 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.066+0000 7f68a4ff9640 1 -- 172.21.15.155:0/1518028703 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4258308409 0 0) 0x7f68a8139990 con 0x7f68a813f710 2024-03-25T18:33:15.068 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.067+0000 7f68a4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:33:15.068 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.067+0000 7f68a4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:15.068 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.067+0000 7f68a4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:15.068 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.067+0000 7f68a4ff9640 10 monclient(hunting): my global_id is 4619 2024-03-25T18:33:15.068 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.067+0000 7f68a4ff9640 1 -- 172.21.15.155:0/1518028703 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6894004eb0 con 0x7f68a813f710 2024-03-25T18:33:15.068 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.067+0000 7f68a4ff9640 1 -- 172.21.15.155:0/1518028703 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1539056207 0 0) 0x7f68a814e7d0 con 0x7f68a81430c0 2024-03-25T18:33:15.068 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.067+0000 7f68a4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:33:15.068 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.067+0000 7f68a4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:15.068 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.067+0000 7f68a4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:15.069 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.067+0000 7f68a4ff9640 10 monclient(hunting): my global_id is 4606 2024-03-25T18:33:15.069 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.067+0000 7f68a4ff9640 1 -- 172.21.15.155:0/1518028703 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68a8139990 con 0x7f68a81430c0 2024-03-25T18:33:15.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.070+0000 7f68a4ff9640 1 -- 172.21.15.155:0/1518028703 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 773150619 0 0) 0x7f68a814e580 con 0x7f68a80a98f0 2024-03-25T18:33:15.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.070+0000 7f68a4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:33:15.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.070+0000 7f68a4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:15.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.070+0000 7f68a4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:15.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.070+0000 7f68a4ff9640 10 monclient(hunting): my global_id is 4611 2024-03-25T18:33:15.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.070+0000 7f68a4ff9640 1 -- 172.21.15.155:0/1518028703 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68a814e7d0 con 0x7f68a80a98f0 2024-03-25T18:33:15.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.075+0000 7f68a4ff9640 1 -- 172.21.15.155:0/1518028703 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3673766700 0 0) 0x7f68a8139990 con 0x7f68a81430c0 2024-03-25T18:33:15.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.076+0000 7f68a4ff9640 1 -- 172.21.15.155:0/1518028703 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f68a814e580 con 0x7f68a81430c0 2024-03-25T18:33:15.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.076+0000 7f68a4ff9640 1 -- 172.21.15.155:0/1518028703 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f68a0001e80 con 0x7f68a81430c0 2024-03-25T18:33:15.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.076+0000 7f68a4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:15.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.076+0000 7f68a4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:33:15.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.076+0000 7f68a4ff9640 10 monclient(hunting): dump: 2024-03-25T18:33:15.078 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:33:15.078 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:33:15.078 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:15.078 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:15.078 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:15.078 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:33:15.078 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:33:15.078 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:33:15.078 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:33:15.078 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:33:15.078 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.076+0000 7f68a4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:33:15.086 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.085+0000 7f68a4ff9640 1 -- 172.21.15.155:0/1518028703 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 293689005 0 0) 0x7f6894004eb0 con 0x7f68a813f710 2024-03-25T18:33:15.087 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.085+0000 7f68a4ff9640 1 -- 172.21.15.155:0/1518028703 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f68a8139990 con 0x7f68a813f710 2024-03-25T18:33:15.087 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.085+0000 7f68a4ff9640 1 -- 172.21.15.155:0/1518028703 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f689c001910 con 0x7f68a813f710 2024-03-25T18:33:15.087 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.085+0000 7f68a4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:15.087 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.085+0000 7f68a4ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:33:15.087 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.085+0000 7f68a4ff9640 10 monclient(hunting): dump: 2024-03-25T18:33:15.087 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:33:15.087 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:33:15.087 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:15.087 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:15.087 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:15.087 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:33:15.087 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:33:15.087 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:33:15.088 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:33:15.088 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:33:15.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.090+0000 7f68a4ff9640 1 -- 172.21.15.155:0/1518028703 <== mon.1 v1:172.21.15.167:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1491568425 0 0) 0x7f68a814e580 con 0x7f68a81430c0 2024-03-25T18:33:15.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.090+0000 7f68a4ff9640 1 -- 172.21.15.155:0/1518028703 >> v1:172.21.15.155:6790/0 conn(0x7f68a813f710 legacy=0x7f68a813a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:15.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.090+0000 7f68a4ff9640 1 -- 172.21.15.155:0/1518028703 >> v1:172.21.15.155:6789/0 conn(0x7f68a80a98f0 legacy=0x7f68a813c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:15.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.090+0000 7f68a4ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:33:15.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.090+0000 7f68a4ff9640 1 monclient: found mon.b 2024-03-25T18:33:15.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.090+0000 7f68a4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:15.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.090+0000 7f68a4ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:33:15.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.090+0000 7f68a4ff9640 1 -- 172.21.15.155:0/1518028703 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68a813bf00 con 0x7f68a81430c0 2024-03-25T18:33:15.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.090+0000 7f68a4ff9640 10 monclient: _finish_auth 0 2024-03-25T18:33:15.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.090+0000 7f68a4ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:33:15.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.090+0000 7f68a4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:15.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.090+0000 7f68ae38f640 5 monclient: authenticate success, global_id 4606 2024-03-25T18:33:15.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.090+0000 7f68ae38f640 10 monclient: _renew_subs 2024-03-25T18:33:15.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.091+0000 7f68ae38f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:33:15.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.091+0000 7f68ae38f640 1 -- 172.21.15.155:0/1518028703 --> v1:172.21.15.167:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68a813c1e0 con 0x7f68a81430c0 2024-03-25T18:33:15.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.091+0000 7f68ae38f640 10 client.4606.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:33:15.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.091+0000 7f68ae38f640 10 monclient: _renew_subs 2024-03-25T18:33:15.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.091+0000 7f68ae38f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:33:15.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.091+0000 7f68ae38f640 1 -- 172.21.15.155:0/1518028703 --> v1:172.21.15.167:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f68a813b070 con 0x7f68a81430c0 2024-03-25T18:33:15.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.091+0000 7f68ae38f640 1 librados: init done 2024-03-25T18:33:15.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.091+0000 7f68ae38f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:15.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.092+0000 7f68ae38f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:15.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.092+0000 7f68ae38f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:33:15.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.092+0000 7f68ae38f640 1 -- 172.21.15.155:0/1518028703 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68a80627b0 con 0x7f68a81430c0 2024-03-25T18:33:15.113 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.112+0000 7f68a4ff9640 1 -- 172.21.15.155:0/1518028703 <== mon.1 v1:172.21.15.167:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f68a0001fe0 con 0x7f68a81430c0 2024-03-25T18:33:15.113 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.112+0000 7f68a4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:15.113 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.112+0000 7f68a4ff9640 1 -- 172.21.15.155:0/1518028703 <== mon.1 v1:172.21.15.167:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f68a0003dc0 con 0x7f68a81430c0 2024-03-25T18:33:15.113 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.112+0000 7f68a4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:15.113 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.112+0000 7f68a4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:33:15.113 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.112+0000 7f68a4ff9640 10 monclient: dump: 2024-03-25T18:33:15.114 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:33:15.114 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:33:15.114 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:15.114 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:15.114 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:15.114 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:33:15.114 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:33:15.114 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:33:15.114 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:33:15.114 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:33:15.114 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.112+0000 7f68a67fc640 4 set_mon_vals no callback set 2024-03-25T18:33:15.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.113+0000 7f68a4ff9640 1 -- 172.21.15.155:0/1518028703 <== mon.1 v1:172.21.15.167:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7f68a0019d90 con 0x7f68a81430c0 2024-03-25T18:33:15.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.114+0000 7f68a7fff640 10 monclient: get_auth_request con 0x7f689406dde0 auth_method 0 2024-03-25T18:33:15.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.114+0000 7f68a4ff9640 10 client.4606.objecter ms_handle_connect 0x7f689406dde0 2024-03-25T18:33:15.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.116+0000 7f68a4ff9640 1 -- 172.21.15.155:0/1518028703 <== mon.1 v1:172.21.15.167:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7f68a0053970 con 0x7f68a81430c0 2024-03-25T18:33:15.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.116+0000 7f68a4ff9640 10 client.4606.objecter ms_dispatch 0x7f68a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:33:15.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.116+0000 7f68a4ff9640 3 client.4606.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:33:15.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.116+0000 7f68a4ff9640 3 client.4606.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:33:15.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.116+0000 7f68a4ff9640 20 client.4606.objecter dump_active .. 0 homeless 2024-03-25T18:33:15.391 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.389+0000 7f68a4ff9640 1 -- 172.21.15.155:0/1518028703 <== mon.1 v1:172.21.15.167:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f68a004f480 con 0x7f68a81430c0 2024-03-25T18:33:15.391 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.389+0000 7f68a4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:15.391 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.389+0000 7f68a4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:33:15.609 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.608+0000 7f68ae38f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:15.610 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.608+0000 7f68ae38f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:15.610 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.608+0000 7f68ae38f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:33:15.610 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.608+0000 7f68ae38f640 1 -- 172.21.15.155:0/1518028703 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f68a813e230 con 0x7f68a81430c0 2024-03-25T18:33:15.642 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.640+0000 7f68a4ff9640 1 -- 172.21.15.155:0/1518028703 <== mon.1 v1:172.21.15.167:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7f68a00513b0 con 0x7f68a81430c0 2024-03-25T18:33:15.642 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.640+0000 7f68a4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:15.642 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.640+0000 7f68a4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:33:15.642 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:33:15.642 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:33:15.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.643+0000 7f68ae38f640 10 librados: watch_flush enter 2024-03-25T18:33:15.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.643+0000 7f68ae38f640 10 librados: watch_flush exit 2024-03-25T18:33:15.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.643+0000 7f68ae38f640 20 client.4606.objecter shutdown clearing up homeless session... 2024-03-25T18:33:15.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.643+0000 7f68ae38f640 10 client.4606.objecter successfully canceled tick 2024-03-25T18:33:15.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.643+0000 7f68ae38f640 1 -- 172.21.15.155:0/1518028703 >> v1:172.21.15.167:6800/2919070503 conn(0x7f689406dde0 legacy=0x7f68940702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:15.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.643+0000 7f68ae38f640 10 monclient: shutdown 2024-03-25T18:33:15.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.643+0000 7f68ae38f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:15.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.643+0000 7f68ae38f640 1 -- 172.21.15.155:0/1518028703 >> v1:172.21.15.167:6789/0 conn(0x7f68a81430c0 legacy=0x7f68a81e66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:15.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.643+0000 7f68ae38f640 1 -- 172.21.15.155:0/1518028703 shutdown_connections 2024-03-25T18:33:15.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.643+0000 7f68ae38f640 1 -- 172.21.15.155:0/1518028703 >> 172.21.15.155:0/1518028703 conn(0x7f68a80b7ea0 msgr2=0x7f68a814ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:15.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.644+0000 7f68ae38f640 1 -- 172.21.15.155:0/1518028703 shutdown_connections 2024-03-25T18:33:15.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.644+0000 7f68ae38f640 1 -- 172.21.15.155:0/1518028703 wait complete. 2024-03-25T18:33:15.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:15.644+0000 7f68ae38f640 1 librados: shutdown 2024-03-25T18:33:15.656 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:33:22.462 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:33:22.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.577+0000 7f93e8298640 10 monclient: get_monmap_and_config 2024-03-25T18:33:22.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.577+0000 7f93e8298640 10 monclient: build_initial_monmap 2024-03-25T18:33:22.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.577+0000 7f93e8298640 10 monclient: monmap: 2024-03-25T18:33:22.578 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:33:22.578 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:33:22.578 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:33:22.578181+0000 2024-03-25T18:33:22.578 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:33:22.578181+0000 2024-03-25T18:33:22.579 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:22.579 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:33:22.579 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:33:22.579 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:33:22.579 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:33:22.579 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:33:22.580 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.577+0000 7f93e8298640 1 Processor -- start 2024-03-25T18:33:22.580 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.577+0000 7f93e8298640 1 -- start start 2024-03-25T18:33:22.580 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.577+0000 7f93e8298640 10 monclient: init 2024-03-25T18:33:22.580 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.578+0000 7f93e8298640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:22.580 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.578+0000 7f93e8298640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:33:22.580 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.578+0000 7f93e8298640 10 monclient(hunting): picked mon.noname-a con 0x7f93e00b4d60 addr v1:172.21.15.155:6789/0 2024-03-25T18:33:22.580 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.578+0000 7f93e8298640 10 monclient(hunting): picked mon.noname-b con 0x7f93e00b1730 addr v1:172.21.15.155:6790/0 2024-03-25T18:33:22.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.578+0000 7f93e8298640 10 monclient(hunting): picked mon.noname-c con 0x7f93e014be90 addr v1:172.21.15.167:6789/0 2024-03-25T18:33:22.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.578+0000 7f93e8298640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93e00af940 con 0x7f93e00b4d60 2024-03-25T18:33:22.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.578+0000 7f93e8298640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93e00b0a70 con 0x7f93e014be90 2024-03-25T18:33:22.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.578+0000 7f93e8298640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93e00b0d50 con 0x7f93e00b1730 2024-03-25T18:33:22.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.578+0000 7f93e8298640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:22.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.578+0000 7f93e8298640 10 monclient(hunting): authenticate will time out at 8048.888664s 2024-03-25T18:33:22.589 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.588+0000 7f93e6835640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f93e014be90 0x7f93e014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:39082/0 (socket says 172.21.15.155:39082) 2024-03-25T18:33:22.589 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.588+0000 7f93e6835640 1 -- 172.21.15.155:0/3634606189 learned_addr learned my addr 172.21.15.155:0/3634606189 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:33:22.600 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.598+0000 7f93e5032640 1 -- 172.21.15.155:0/3634606189 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3230472141 0 0) 0x7f93e00b0d50 con 0x7f93e00b1730 2024-03-25T18:33:22.600 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.598+0000 7f93e5032640 10 monclient(hunting): _negotiate 2024-03-25T18:33:22.600 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.598+0000 7f93e5032640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:22.600 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.598+0000 7f93e5032640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:22.600 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.598+0000 7f93e5032640 10 monclient(hunting): my global_id is 4622 2024-03-25T18:33:22.600 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.598+0000 7f93e5032640 1 -- 172.21.15.155:0/3634606189 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93cc003520 con 0x7f93e00b1730 2024-03-25T18:33:22.607 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.606+0000 7f93e5032640 1 -- 172.21.15.155:0/3634606189 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3213243924 0 0) 0x7f93e00af940 con 0x7f93e00b4d60 2024-03-25T18:33:22.607 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.606+0000 7f93e5032640 10 monclient(hunting): _negotiate 2024-03-25T18:33:22.607 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.606+0000 7f93e5032640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:22.607 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.606+0000 7f93e5032640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:22.607 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.606+0000 7f93e5032640 10 monclient(hunting): my global_id is 4614 2024-03-25T18:33:22.608 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.606+0000 7f93e5032640 1 -- 172.21.15.155:0/3634606189 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93e00b0d50 con 0x7f93e00b4d60 2024-03-25T18:33:22.608 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.607+0000 7f93e5032640 1 -- 172.21.15.155:0/3634606189 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1174425900 0 0) 0x7f93e00b0a70 con 0x7f93e014be90 2024-03-25T18:33:22.608 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.607+0000 7f93e5032640 10 monclient(hunting): _negotiate 2024-03-25T18:33:22.608 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.607+0000 7f93e5032640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:22.609 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.607+0000 7f93e5032640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:22.609 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.607+0000 7f93e5032640 10 monclient(hunting): my global_id is 4609 2024-03-25T18:33:22.609 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.607+0000 7f93e5032640 1 -- 172.21.15.155:0/3634606189 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93e00af940 con 0x7f93e014be90 2024-03-25T18:33:22.611 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.610+0000 7f93e5032640 1 -- 172.21.15.155:0/3634606189 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1493726028 0 0) 0x7f93cc003520 con 0x7f93e00b1730 2024-03-25T18:33:22.611 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.610+0000 7f93e5032640 1 -- 172.21.15.155:0/3634606189 >> v1:172.21.15.167:6789/0 conn(0x7f93e014be90 legacy=0x7f93e014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:22.611 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.610+0000 7f93e5032640 1 -- 172.21.15.155:0/3634606189 >> v1:172.21.15.155:6789/0 conn(0x7f93e00b4d60 legacy=0x7f93e013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:22.611 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.610+0000 7f93e5032640 10 monclient: _finish_hunting 0 2024-03-25T18:33:22.612 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.610+0000 7f93e5032640 1 monclient: found mon.noname-b 2024-03-25T18:33:22.612 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.610+0000 7f93e5032640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:22.612 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.610+0000 7f93e5032640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-25T18:33:22.612 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.610+0000 7f93e5032640 1 -- 172.21.15.155:0/3634606189 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93e00b1030 con 0x7f93e00b1730 2024-03-25T18:33:22.612 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.610+0000 7f93e5032640 10 monclient: _finish_auth 0 2024-03-25T18:33:22.612 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.610+0000 7f93e5032640 10 monclient: _check_auth_tickets 2024-03-25T18:33:22.612 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.610+0000 7f93e5032640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:22.612 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.610+0000 7f93e5032640 1 -- 172.21.15.155:0/3634606189 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f93d0001690 con 0x7f93e00b1730 2024-03-25T18:33:22.612 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.610+0000 7f93e5032640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:22.612 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.610+0000 7f93e5032640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:33:22.612 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.610+0000 7f93e5032640 10 monclient: dump: 2024-03-25T18:33:22.612 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:33:22.612 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:33:22.612 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:22.612 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:22.613 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:22.613 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:33:22.613 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:33:22.613 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:33:22.613 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:33:22.613 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:33:22.613 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.610+0000 7f93e8298640 5 monclient: authenticate success, global_id 4622 2024-03-25T18:33:22.613 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.610+0000 7f93e8298640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:22.623 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.622+0000 7f93e5032640 1 -- 172.21.15.155:0/3634606189 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f93d00017f0 con 0x7f93e00b1730 2024-03-25T18:33:22.623 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.622+0000 7f93e5032640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:22.623 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.622+0000 7f93e5032640 1 -- 172.21.15.155:0/3634606189 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f93d0001c00 con 0x7f93e00b1730 2024-03-25T18:33:22.623 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.622+0000 7f93e8298640 10 monclient: get_monmap_and_config success 2024-03-25T18:33:22.623 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.622+0000 7f93e8298640 4 set_mon_vals no callback set 2024-03-25T18:33:22.623 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.622+0000 7f93e5032640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:22.623 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.622+0000 7f93e5032640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:33:22.623 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.622+0000 7f93e5032640 10 monclient: dump: 2024-03-25T18:33:22.624 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:33:22.624 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:33:22.624 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:22.624 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:22.624 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:22.624 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:33:22.624 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:33:22.624 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:33:22.624 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:33:22.624 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:33:22.626 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.622+0000 7f93e8298640 10 monclient: shutdown 2024-03-25T18:33:22.626 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.622+0000 7f93e8298640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:22.626 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.622+0000 7f93e8298640 1 -- 172.21.15.155:0/3634606189 >> v1:172.21.15.155:6790/0 conn(0x7f93e00b1730 legacy=0x7f93e00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:22.626 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.623+0000 7f93e8298640 1 -- 172.21.15.155:0/3634606189 shutdown_connections 2024-03-25T18:33:22.626 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.623+0000 7f93e8298640 1 -- 172.21.15.155:0/3634606189 >> 172.21.15.155:0/3634606189 conn(0x7f93e00617a0 msgr2=0x7f93e0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:22.627 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.623+0000 7f93e8298640 1 -- 172.21.15.155:0/3634606189 shutdown_connections 2024-03-25T18:33:22.627 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.623+0000 7f93e8298640 1 -- 172.21.15.155:0/3634606189 wait complete. 2024-03-25T18:33:22.627 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.623+0000 7f93e8298640 10 monclient: build_initial_monmap 2024-03-25T18:33:22.627 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.623+0000 7f93e8298640 10 monclient: monmap: 2024-03-25T18:33:22.627 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:33:22.627 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:33:22.627 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:33:22.627 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:33:22.627 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:22.627 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:33:22.627 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:33:22.627 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:33:22.627 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:33:22.627 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:33:22.628 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.624+0000 7f93e8298640 1 librados: starting msgr at 2024-03-25T18:33:22.628 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.624+0000 7f93e8298640 1 librados: starting objecter 2024-03-25T18:33:22.628 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.624+0000 7f93e8298640 1 Processor -- start 2024-03-25T18:33:22.628 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.624+0000 7f93e8298640 1 -- start start 2024-03-25T18:33:22.628 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.624+0000 7f93e8298640 1 librados: setting wanted keys 2024-03-25T18:33:22.628 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.624+0000 7f93e8298640 1 librados: calling monclient init 2024-03-25T18:33:22.628 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.624+0000 7f93e8298640 10 monclient: init 2024-03-25T18:33:22.628 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.625+0000 7f93e8298640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:22.628 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.625+0000 7f93e8298640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:33:22.628 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.625+0000 7f93e8298640 10 monclient(hunting): picked mon.noname-a con 0x7f93e00b1730 addr v1:172.21.15.155:6789/0 2024-03-25T18:33:22.628 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.625+0000 7f93e8298640 10 monclient(hunting): picked mon.noname-c con 0x7f93e00b4d60 addr v1:172.21.15.155:6790/0 2024-03-25T18:33:22.628 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.625+0000 7f93e8298640 10 monclient(hunting): picked mon.noname-b con 0x7f93e014be90 addr v1:172.21.15.167:6789/0 2024-03-25T18:33:22.628 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.625+0000 7f93e8298640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93e0143900 con 0x7f93e00b1730 2024-03-25T18:33:22.628 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.625+0000 7f93e8298640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93e013a550 con 0x7f93e014be90 2024-03-25T18:33:22.628 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.625+0000 7f93e8298640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93e013bc40 con 0x7f93e00b4d60 2024-03-25T18:33:22.629 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.625+0000 7f93e8298640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:22.629 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.625+0000 7f93e8298640 10 monclient(hunting): authenticate will time out at 8048.934913s 2024-03-25T18:33:22.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.636+0000 7f93e6034640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f93e00b1730 0x7f93e0139e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:43326/0 (socket says 172.21.15.155:43326) 2024-03-25T18:33:22.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.636+0000 7f93e6034640 1 -- 172.21.15.155:0/2485340810 learned_addr learned my addr 172.21.15.155:0/2485340810 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:33:22.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.638+0000 7f93d6ffd640 10 client.?.objecter ms_handle_connect 0x7f93e00b1730 2024-03-25T18:33:22.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.638+0000 7f93d6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:22.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.638+0000 7f93d6ffd640 10 client.?.objecter ms_handle_connect 0x7f93e014be90 2024-03-25T18:33:22.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.638+0000 7f93d6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:22.642 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.641+0000 7f93d6ffd640 10 client.?.objecter ms_handle_connect 0x7f93e00b4d60 2024-03-25T18:33:22.642 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.641+0000 7f93d6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:22.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.646+0000 7f93d6ffd640 1 -- 172.21.15.155:0/2485340810 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 793564056 0 0) 0x7f93e0143900 con 0x7f93e00b1730 2024-03-25T18:33:22.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.646+0000 7f93d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:22.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.646+0000 7f93d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:22.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.646+0000 7f93d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:22.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.646+0000 7f93d6ffd640 10 monclient(hunting): my global_id is 4617 2024-03-25T18:33:22.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.646+0000 7f93d6ffd640 1 -- 172.21.15.155:0/2485340810 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93c4004eb0 con 0x7f93e00b1730 2024-03-25T18:33:22.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.646+0000 7f93d6ffd640 1 -- 172.21.15.155:0/2485340810 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2145748282 0 0) 0x7f93e013bc40 con 0x7f93e00b4d60 2024-03-25T18:33:22.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.646+0000 7f93d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:22.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.646+0000 7f93d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:22.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.646+0000 7f93d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:22.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.646+0000 7f93d6ffd640 10 monclient(hunting): my global_id is 4625 2024-03-25T18:33:22.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.646+0000 7f93d6ffd640 1 -- 172.21.15.155:0/2485340810 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93e0143900 con 0x7f93e00b4d60 2024-03-25T18:33:22.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.648+0000 7f93d6ffd640 1 -- 172.21.15.155:0/2485340810 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1918005904 0 0) 0x7f93e013a550 con 0x7f93e014be90 2024-03-25T18:33:22.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.648+0000 7f93d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:22.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.648+0000 7f93d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:22.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.648+0000 7f93d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:22.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.648+0000 7f93d6ffd640 10 monclient(hunting): my global_id is 4612 2024-03-25T18:33:22.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.648+0000 7f93d6ffd640 1 -- 172.21.15.155:0/2485340810 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93e013bc40 con 0x7f93e014be90 2024-03-25T18:33:22.654 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.653+0000 7f93d6ffd640 1 -- 172.21.15.155:0/2485340810 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2049028180 0 0) 0x7f93c4004eb0 con 0x7f93e00b1730 2024-03-25T18:33:22.654 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.653+0000 7f93d6ffd640 1 -- 172.21.15.155:0/2485340810 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f93e013a550 con 0x7f93e00b1730 2024-03-25T18:33:22.654 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.653+0000 7f93d6ffd640 1 -- 172.21.15.155:0/2485340810 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f93dc001a30 con 0x7f93e00b1730 2024-03-25T18:33:22.654 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.653+0000 7f93d6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:22.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.653+0000 7f93d6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:33:22.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.653+0000 7f93d6ffd640 10 monclient(hunting): dump: 2024-03-25T18:33:22.655 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:33:22.655 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:33:22.655 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:22.655 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:22.655 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:22.655 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:33:22.655 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:33:22.655 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:33:22.655 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:33:22.655 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:33:22.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.653+0000 7f93d6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:33:22.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.657+0000 7f93d6ffd640 1 -- 172.21.15.155:0/2485340810 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 204602222 0 0) 0x7f93e0143900 con 0x7f93e00b4d60 2024-03-25T18:33:22.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.658+0000 7f93d6ffd640 1 -- 172.21.15.155:0/2485340810 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f93c4004eb0 con 0x7f93e00b4d60 2024-03-25T18:33:22.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.658+0000 7f93d6ffd640 1 -- 172.21.15.155:0/2485340810 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f93d0001b00 con 0x7f93e00b4d60 2024-03-25T18:33:22.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.658+0000 7f93d6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:22.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.658+0000 7f93d6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:33:22.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.658+0000 7f93d6ffd640 10 monclient(hunting): dump: 2024-03-25T18:33:22.659 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:33:22.659 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:33:22.659 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:22.660 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:22.660 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:22.660 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:33:22.660 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:33:22.660 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:33:22.660 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:33:22.660 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:33:22.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.663+0000 7f93d6ffd640 1 -- 172.21.15.155:0/2485340810 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1400151260 0 0) 0x7f93e013a550 con 0x7f93e00b1730 2024-03-25T18:33:22.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.664+0000 7f93d6ffd640 1 -- 172.21.15.155:0/2485340810 >> v1:172.21.15.155:6790/0 conn(0x7f93e00b4d60 legacy=0x7f93e012fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:22.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.664+0000 7f93d6ffd640 1 -- 172.21.15.155:0/2485340810 >> v1:172.21.15.167:6789/0 conn(0x7f93e014be90 legacy=0x7f93e01d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:22.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.664+0000 7f93d6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:33:22.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.664+0000 7f93d6ffd640 1 monclient: found mon.a 2024-03-25T18:33:22.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.664+0000 7f93d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:22.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.664+0000 7f93d6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:33:22.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.664+0000 7f93d6ffd640 1 -- 172.21.15.155:0/2485340810 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93e013bdd0 con 0x7f93e00b1730 2024-03-25T18:33:22.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.664+0000 7f93d6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:33:22.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.664+0000 7f93d6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:33:22.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.664+0000 7f93d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:22.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.664+0000 7f93e8298640 5 monclient: authenticate success, global_id 4617 2024-03-25T18:33:22.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.664+0000 7f93e8298640 10 monclient: _renew_subs 2024-03-25T18:33:22.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.664+0000 7f93e8298640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:33:22.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.664+0000 7f93e8298640 1 -- 172.21.15.155:0/2485340810 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93e01391c0 con 0x7f93e00b1730 2024-03-25T18:33:22.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.664+0000 7f93e8298640 10 client.4617.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:33:22.667 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.664+0000 7f93e8298640 10 monclient: _renew_subs 2024-03-25T18:33:22.667 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.664+0000 7f93e8298640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:33:22.667 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.664+0000 7f93e8298640 1 -- 172.21.15.155:0/2485340810 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f93e0130670 con 0x7f93e00b1730 2024-03-25T18:33:22.667 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.664+0000 7f93e8298640 1 librados: init done 2024-03-25T18:33:22.667 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.665+0000 7f93e8298640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:22.667 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.665+0000 7f93e8298640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:22.667 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.665+0000 7f93e8298640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:33:22.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.665+0000 7f93e8298640 1 -- 172.21.15.155:0/2485340810 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93e01d7a80 con 0x7f93e00b1730 2024-03-25T18:33:22.963 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.962+0000 7f93d6ffd640 1 -- 172.21.15.155:0/2485340810 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f93dc001c70 con 0x7f93e00b1730 2024-03-25T18:33:22.963 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.962+0000 7f93d6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:22.963 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.962+0000 7f93d6ffd640 1 -- 172.21.15.155:0/2485340810 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f93dc003a60 con 0x7f93e00b1730 2024-03-25T18:33:22.964 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.962+0000 7f93d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:22.964 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.962+0000 7f93d6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:33:22.964 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.962+0000 7f93d6ffd640 10 monclient: dump: 2024-03-25T18:33:22.964 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:33:22.964 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:33:22.964 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:22.964 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:22.964 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:22.964 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:33:22.964 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:33:22.964 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:33:22.964 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:33:22.964 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:33:22.964 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.962+0000 7f93e4831640 4 set_mon_vals no callback set 2024-03-25T18:33:22.965 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.963+0000 7f93d6ffd640 1 -- 172.21.15.155:0/2485340810 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7f93dc003c80 con 0x7f93e00b1730 2024-03-25T18:33:22.965 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.964+0000 7f93e5833640 10 monclient: get_auth_request con 0x7f93c406df70 auth_method 0 2024-03-25T18:33:22.965 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.964+0000 7f93d6ffd640 10 client.4617.objecter ms_handle_connect 0x7f93c406df70 2024-03-25T18:33:22.972 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.971+0000 7f93d6ffd640 1 -- 172.21.15.155:0/2485340810 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7f93dc054870 con 0x7f93e00b1730 2024-03-25T18:33:22.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.971+0000 7f93d6ffd640 10 client.4617.objecter ms_dispatch 0x7f93e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:33:22.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.971+0000 7f93d6ffd640 3 client.4617.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:33:22.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.971+0000 7f93d6ffd640 3 client.4617.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:33:22.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.971+0000 7f93d6ffd640 20 client.4617.objecter dump_active .. 0 homeless 2024-03-25T18:33:22.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.971+0000 7f93d6ffd640 1 -- 172.21.15.155:0/2485340810 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f93dc050380 con 0x7f93e00b1730 2024-03-25T18:33:22.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.971+0000 7f93d6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:22.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:22.971+0000 7f93d6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:33:23.224 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:23.223+0000 7f93e8298640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:23.224 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:23.223+0000 7f93e8298640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:23.224 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:23.223+0000 7f93e8298640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:33:23.224 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:23.223+0000 7f93e8298640 1 -- 172.21.15.155:0/2485340810 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f93e00627b0 con 0x7f93e00b1730 2024-03-25T18:33:23.250 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:23.248+0000 7f93d6ffd640 1 -- 172.21.15.155:0/2485340810 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7f93dc0522b0 con 0x7f93e00b1730 2024-03-25T18:33:23.250 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:23.248+0000 7f93d6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:23.250 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:23.248+0000 7f93d6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:33:23.250 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:33:23.250 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:33:23.253 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:23.250+0000 7f93e8298640 10 librados: watch_flush enter 2024-03-25T18:33:23.253 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:23.251+0000 7f93e8298640 10 librados: watch_flush exit 2024-03-25T18:33:23.253 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:23.251+0000 7f93e8298640 20 client.4617.objecter shutdown clearing up homeless session... 2024-03-25T18:33:23.253 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:23.251+0000 7f93e8298640 10 client.4617.objecter successfully canceled tick 2024-03-25T18:33:23.253 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:23.251+0000 7f93e8298640 1 -- 172.21.15.155:0/2485340810 >> v1:172.21.15.167:6800/2919070503 conn(0x7f93c406df70 legacy=0x7f93c4070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:23.253 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:23.251+0000 7f93e8298640 10 monclient: shutdown 2024-03-25T18:33:23.253 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:23.251+0000 7f93e8298640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:23.253 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:23.251+0000 7f93e8298640 1 -- 172.21.15.155:0/2485340810 >> v1:172.21.15.155:6789/0 conn(0x7f93e00b1730 legacy=0x7f93e0139e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:23.253 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:23.251+0000 7f93e8298640 1 -- 172.21.15.155:0/2485340810 shutdown_connections 2024-03-25T18:33:23.253 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:23.251+0000 7f93e8298640 1 -- 172.21.15.155:0/2485340810 >> 172.21.15.155:0/2485340810 conn(0x7f93e00617a0 msgr2=0x7f93e0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:23.253 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:23.251+0000 7f93e8298640 1 -- 172.21.15.155:0/2485340810 shutdown_connections 2024-03-25T18:33:23.253 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:23.251+0000 7f93e8298640 1 -- 172.21.15.155:0/2485340810 wait complete. 2024-03-25T18:33:23.253 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:23.251+0000 7f93e8298640 1 librados: shutdown 2024-03-25T18:33:23.262 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:33:30.071 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:33:30.187 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.184+0000 7f7634ce4640 10 monclient: get_monmap_and_config 2024-03-25T18:33:30.187 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.184+0000 7f7634ce4640 10 monclient: build_initial_monmap 2024-03-25T18:33:30.187 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.184+0000 7f7634ce4640 10 monclient: monmap: 2024-03-25T18:33:30.188 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:33:30.188 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:33:30.188 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:33:30.186702+0000 2024-03-25T18:33:30.188 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:33:30.186702+0000 2024-03-25T18:33:30.188 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:30.188 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:33:30.188 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:33:30.188 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:33:30.188 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:33:30.188 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:33:30.188 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.185+0000 7f7634ce4640 1 Processor -- start 2024-03-25T18:33:30.188 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.185+0000 7f7634ce4640 1 -- start start 2024-03-25T18:33:30.188 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.185+0000 7f7634ce4640 10 monclient: init 2024-03-25T18:33:30.189 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.186+0000 7f7634ce4640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:30.189 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.186+0000 7f7634ce4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:33:30.189 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.186+0000 7f7634ce4640 10 monclient(hunting): picked mon.noname-c con 0x7f76300a98f0 addr v1:172.21.15.167:6789/0 2024-03-25T18:33:30.190 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.186+0000 7f7634ce4640 10 monclient(hunting): picked mon.noname-b con 0x7f763013f710 addr v1:172.21.15.155:6790/0 2024-03-25T18:33:30.190 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.186+0000 7f7634ce4640 10 monclient(hunting): picked mon.noname-a con 0x7f76301430c0 addr v1:172.21.15.155:6789/0 2024-03-25T18:33:30.190 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.186+0000 7f7634ce4640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7630142520 con 0x7f76301430c0 2024-03-25T18:33:30.190 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.186+0000 7f7634ce4640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76300b28a0 con 0x7f76300a98f0 2024-03-25T18:33:30.190 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.186+0000 7f7634ce4640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76300b2a30 con 0x7f763013f710 2024-03-25T18:33:30.190 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.186+0000 7f7634ce4640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:30.190 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.186+0000 7f7634ce4640 10 monclient(hunting): authenticate will time out at 8056.497271s 2024-03-25T18:33:30.206 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.204+0000 7f762e59c640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f76300a98f0 0x7f76300a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:43924/0 (socket says 172.21.15.155:43924) 2024-03-25T18:33:30.206 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.204+0000 7f762e59c640 1 -- 172.21.15.155:0/3328897992 learned_addr learned my addr 172.21.15.155:0/3328897992 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:33:30.218 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.215+0000 7f762dd9b640 1 -- 172.21.15.155:0/3328897992 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 501302078 0 0) 0x7f76300b28a0 con 0x7f76300a98f0 2024-03-25T18:33:30.218 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.215+0000 7f762dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:33:30.218 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.215+0000 7f762dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:30.218 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.215+0000 7f762dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:30.218 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.215+0000 7f762dd9b640 10 monclient(hunting): my global_id is 4615 2024-03-25T18:33:30.218 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.215+0000 7f762dd9b640 1 -- 172.21.15.155:0/3328897992 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f761c003520 con 0x7f76300a98f0 2024-03-25T18:33:30.226 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.223+0000 7f762dd9b640 1 -- 172.21.15.155:0/3328897992 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 542376158 0 0) 0x7f761c003520 con 0x7f76300a98f0 2024-03-25T18:33:30.226 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.223+0000 7f762dd9b640 1 -- 172.21.15.155:0/3328897992 >> v1:172.21.15.155:6790/0 conn(0x7f763013f710 legacy=0x7f76301429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:30.226 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.223+0000 7f762dd9b640 1 -- 172.21.15.155:0/3328897992 >> v1:172.21.15.155:6789/0 conn(0x7f76301430c0 legacy=0x7f763014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:30.226 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.223+0000 7f762dd9b640 10 monclient: _finish_hunting 0 2024-03-25T18:33:30.226 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.223+0000 7f762dd9b640 1 monclient: found mon.noname-c 2024-03-25T18:33:30.226 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.223+0000 7f762dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:30.226 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.223+0000 7f762dd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.167:6789/0 2024-03-25T18:33:30.226 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.223+0000 7f762dd9b640 1 -- 172.21.15.155:0/3328897992 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76300b44c0 con 0x7f76300a98f0 2024-03-25T18:33:30.226 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.223+0000 7f762dd9b640 10 monclient: _finish_auth 0 2024-03-25T18:33:30.226 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.223+0000 7f762dd9b640 10 monclient: _check_auth_tickets 2024-03-25T18:33:30.226 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.223+0000 7f762dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:30.227 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.223+0000 7f762dd9b640 1 -- 172.21.15.155:0/3328897992 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f7620001690 con 0x7f76300a98f0 2024-03-25T18:33:30.227 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.223+0000 7f762dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:30.227 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.223+0000 7f762dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:33:30.227 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.223+0000 7f762dd9b640 10 monclient: dump: 2024-03-25T18:33:30.227 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:33:30.227 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:33:30.227 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:30.227 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:30.227 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:30.227 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:33:30.227 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:33:30.227 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:33:30.227 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:33:30.227 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:33:30.227 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.223+0000 7f7634ce4640 5 monclient: authenticate success, global_id 4615 2024-03-25T18:33:30.228 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.223+0000 7f7634ce4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:30.233 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.231+0000 7f762dd9b640 1 -- 172.21.15.155:0/3328897992 <== mon.1 v1:172.21.15.167:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76200017f0 con 0x7f76300a98f0 2024-03-25T18:33:30.233 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.231+0000 7f762dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:30.233 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.231+0000 7f762dd9b640 1 -- 172.21.15.155:0/3328897992 <== mon.1 v1:172.21.15.167:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f7620001c40 con 0x7f76300a98f0 2024-03-25T18:33:30.233 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.231+0000 7f7634ce4640 10 monclient: get_monmap_and_config success 2024-03-25T18:33:30.233 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.231+0000 7f7634ce4640 4 set_mon_vals no callback set 2024-03-25T18:33:30.233 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.231+0000 7f762dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:30.233 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.231+0000 7f762dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:33:30.233 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.231+0000 7f762dd9b640 10 monclient: dump: 2024-03-25T18:33:30.233 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:33:30.233 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:33:30.234 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:30.234 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:30.234 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:30.234 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:33:30.234 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:33:30.234 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:33:30.234 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:33:30.234 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:33:30.236 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.231+0000 7f7634ce4640 10 monclient: shutdown 2024-03-25T18:33:30.236 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.231+0000 7f7634ce4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:30.236 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.231+0000 7f7634ce4640 1 -- 172.21.15.155:0/3328897992 >> v1:172.21.15.167:6789/0 conn(0x7f76300a98f0 legacy=0x7f76300a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:30.236 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.231+0000 7f7634ce4640 1 -- 172.21.15.155:0/3328897992 shutdown_connections 2024-03-25T18:33:30.236 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.231+0000 7f7634ce4640 1 -- 172.21.15.155:0/3328897992 >> 172.21.15.155:0/3328897992 conn(0x7f76300b7ea0 msgr2=0x7f76300b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:30.236 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.231+0000 7f7634ce4640 1 -- 172.21.15.155:0/3328897992 shutdown_connections 2024-03-25T18:33:30.237 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.231+0000 7f7634ce4640 1 -- 172.21.15.155:0/3328897992 wait complete. 2024-03-25T18:33:30.237 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.232+0000 7f7634ce4640 10 monclient: build_initial_monmap 2024-03-25T18:33:30.237 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.232+0000 7f7634ce4640 10 monclient: monmap: 2024-03-25T18:33:30.237 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:33:30.237 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:33:30.237 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:33:30.237 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:33:30.237 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:30.237 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:33:30.237 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:33:30.237 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:33:30.237 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:33:30.237 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:33:30.237 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.233+0000 7f7634ce4640 1 librados: starting msgr at 2024-03-25T18:33:30.237 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.233+0000 7f7634ce4640 1 librados: starting objecter 2024-03-25T18:33:30.238 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.233+0000 7f7634ce4640 1 Processor -- start 2024-03-25T18:33:30.238 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.233+0000 7f7634ce4640 1 -- start start 2024-03-25T18:33:30.238 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.233+0000 7f7634ce4640 1 librados: setting wanted keys 2024-03-25T18:33:30.238 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.233+0000 7f7634ce4640 1 librados: calling monclient init 2024-03-25T18:33:30.238 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.233+0000 7f7634ce4640 10 monclient: init 2024-03-25T18:33:30.238 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.234+0000 7f7634ce4640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:30.238 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.234+0000 7f7634ce4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:33:30.238 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.234+0000 7f7634ce4640 10 monclient(hunting): picked mon.noname-c con 0x7f76300a98f0 addr v1:172.21.15.155:6790/0 2024-03-25T18:33:30.238 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.234+0000 7f7634ce4640 10 monclient(hunting): picked mon.noname-a con 0x7f763013f710 addr v1:172.21.15.155:6789/0 2024-03-25T18:33:30.238 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.234+0000 7f7634ce4640 10 monclient(hunting): picked mon.noname-b con 0x7f76301430c0 addr v1:172.21.15.167:6789/0 2024-03-25T18:33:30.238 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.234+0000 7f7634ce4640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7630138b20 con 0x7f763013f710 2024-03-25T18:33:30.238 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.234+0000 7f7634ce4640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7630137a80 con 0x7f76301430c0 2024-03-25T18:33:30.238 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.234+0000 7f7634ce4640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7630139880 con 0x7f76300a98f0 2024-03-25T18:33:30.238 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.234+0000 7f7634ce4640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:30.239 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.234+0000 7f7634ce4640 10 monclient(hunting): authenticate will time out at 8056.545004s 2024-03-25T18:33:30.244 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.242+0000 7f7627fff640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f763013f710 0x7f7630139170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:34936/0 (socket says 172.21.15.155:34936) 2024-03-25T18:33:30.244 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.242+0000 7f7627fff640 1 -- 172.21.15.155:0/3816374487 learned_addr learned my addr 172.21.15.155:0/3816374487 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:33:30.244 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.242+0000 7f7626ffd640 10 client.?.objecter ms_handle_connect 0x7f763013f710 2024-03-25T18:33:30.245 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.243+0000 7f7626ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:30.254 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.252+0000 7f7626ffd640 10 client.?.objecter ms_handle_connect 0x7f76300a98f0 2024-03-25T18:33:30.254 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.252+0000 7f7626ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:30.256 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.254+0000 7f7626ffd640 1 -- 172.21.15.155:0/3816374487 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3288758340 0 0) 0x7f7630138b20 con 0x7f763013f710 2024-03-25T18:33:30.257 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.254+0000 7f7626ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:30.257 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.254+0000 7f7626ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:30.257 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.254+0000 7f7626ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:30.257 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.254+0000 7f7626ffd640 10 monclient(hunting): my global_id is 4623 2024-03-25T18:33:30.257 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.254+0000 7f7626ffd640 1 -- 172.21.15.155:0/3816374487 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7610004eb0 con 0x7f763013f710 2024-03-25T18:33:30.264 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.262+0000 7f7626ffd640 1 -- 172.21.15.155:0/3816374487 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1808908914 0 0) 0x7f7630139880 con 0x7f76300a98f0 2024-03-25T18:33:30.264 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.262+0000 7f7626ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:30.264 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.262+0000 7f7626ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:30.265 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.262+0000 7f7626ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:30.265 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.262+0000 7f7626ffd640 10 monclient(hunting): my global_id is 4631 2024-03-25T18:33:30.265 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.262+0000 7f7626ffd640 1 -- 172.21.15.155:0/3816374487 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7630138b20 con 0x7f76300a98f0 2024-03-25T18:33:30.265 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.263+0000 7f7626ffd640 10 client.?.objecter ms_handle_connect 0x7f76301430c0 2024-03-25T18:33:30.265 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.263+0000 7f7626ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:30.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.271+0000 7f7626ffd640 1 -- 172.21.15.155:0/3816374487 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3490705439 0 0) 0x7f7630137a80 con 0x7f76301430c0 2024-03-25T18:33:30.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.271+0000 7f7626ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:30.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.271+0000 7f7626ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:30.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.271+0000 7f7626ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:30.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.271+0000 7f7626ffd640 10 monclient(hunting): my global_id is 4618 2024-03-25T18:33:30.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.271+0000 7f7626ffd640 1 -- 172.21.15.155:0/3816374487 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7630139880 con 0x7f76301430c0 2024-03-25T18:33:30.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.272+0000 7f7626ffd640 1 -- 172.21.15.155:0/3816374487 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 460966854 0 0) 0x7f7630138b20 con 0x7f76300a98f0 2024-03-25T18:33:30.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.272+0000 7f7626ffd640 1 -- 172.21.15.155:0/3816374487 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7630137a80 con 0x7f76300a98f0 2024-03-25T18:33:30.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.272+0000 7f7626ffd640 1 -- 172.21.15.155:0/3816374487 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f7620001b00 con 0x7f76300a98f0 2024-03-25T18:33:30.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.272+0000 7f7626ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:30.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.272+0000 7f7626ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:33:30.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.272+0000 7f7626ffd640 10 monclient(hunting): dump: 2024-03-25T18:33:30.275 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:33:30.275 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:33:30.275 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:30.275 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:30.275 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:30.275 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:33:30.275 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:33:30.275 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:33:30.275 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:33:30.275 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:33:30.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.272+0000 7f7626ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:33:30.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.274+0000 7f7626ffd640 1 -- 172.21.15.155:0/3816374487 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1978101114 0 0) 0x7f7610004eb0 con 0x7f763013f710 2024-03-25T18:33:30.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.274+0000 7f7626ffd640 1 -- 172.21.15.155:0/3816374487 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7630138b20 con 0x7f763013f710 2024-03-25T18:33:30.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.274+0000 7f7626ffd640 1 -- 172.21.15.155:0/3816374487 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f7618001a30 con 0x7f763013f710 2024-03-25T18:33:30.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.274+0000 7f7626ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:30.277 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.274+0000 7f7626ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:33:30.277 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.274+0000 7f7626ffd640 10 monclient(hunting): dump: 2024-03-25T18:33:30.277 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:33:30.277 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:33:30.277 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:30.277 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:30.277 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:30.277 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:33:30.277 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:33:30.277 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:33:30.277 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:33:30.277 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:33:30.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.281+0000 7f7626ffd640 1 -- 172.21.15.155:0/3816374487 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3905048339 0 0) 0x7f7630138b20 con 0x7f763013f710 2024-03-25T18:33:30.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.281+0000 7f7626ffd640 1 -- 172.21.15.155:0/3816374487 >> v1:172.21.15.155:6790/0 conn(0x7f76300a98f0 legacy=0x7f7630137370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:30.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.281+0000 7f7626ffd640 1 -- 172.21.15.155:0/3816374487 >> v1:172.21.15.167:6789/0 conn(0x7f76301430c0 legacy=0x7f76301e2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:30.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.281+0000 7f7626ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:33:30.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.281+0000 7f7626ffd640 1 monclient: found mon.a 2024-03-25T18:33:30.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.281+0000 7f7626ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:30.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.281+0000 7f7626ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:33:30.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.281+0000 7f7626ffd640 1 -- 172.21.15.155:0/3816374487 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7630139a10 con 0x7f763013f710 2024-03-25T18:33:30.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.281+0000 7f7626ffd640 10 monclient: _finish_auth 0 2024-03-25T18:33:30.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.281+0000 7f7626ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:33:30.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.281+0000 7f7626ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:30.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.281+0000 7f7634ce4640 5 monclient: authenticate success, global_id 4623 2024-03-25T18:33:30.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.281+0000 7f7634ce4640 10 monclient: _renew_subs 2024-03-25T18:33:30.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.281+0000 7f7634ce4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:33:30.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.281+0000 7f7634ce4640 1 -- 172.21.15.155:0/3816374487 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76301dab00 con 0x7f763013f710 2024-03-25T18:33:30.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.281+0000 7f7634ce4640 10 client.4623.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:33:30.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.281+0000 7f7634ce4640 10 monclient: _renew_subs 2024-03-25T18:33:30.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.281+0000 7f7634ce4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:33:30.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.281+0000 7f7634ce4640 1 -- 172.21.15.155:0/3816374487 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f763013b4f0 con 0x7f763013f710 2024-03-25T18:33:30.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.281+0000 7f7634ce4640 1 librados: init done 2024-03-25T18:33:30.285 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.282+0000 7f7634ce4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:30.285 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.282+0000 7f7634ce4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:30.285 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.282+0000 7f7634ce4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:33:30.285 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.282+0000 7f7634ce4640 1 -- 172.21.15.155:0/3816374487 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76300b3e80 con 0x7f763013f710 2024-03-25T18:33:30.590 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.587+0000 7f7626ffd640 1 -- 172.21.15.155:0/3816374487 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7618001c70 con 0x7f763013f710 2024-03-25T18:33:30.590 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.587+0000 7f7626ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:30.590 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.587+0000 7f7626ffd640 1 -- 172.21.15.155:0/3816374487 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f7618003a60 con 0x7f763013f710 2024-03-25T18:33:30.590 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.587+0000 7f7626ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:30.590 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.587+0000 7f7626ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:33:30.590 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.587+0000 7f7626ffd640 10 monclient: dump: 2024-03-25T18:33:30.591 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:33:30.591 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:33:30.591 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:30.591 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:30.591 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:30.591 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:33:30.591 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:33:30.591 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:33:30.591 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:33:30.591 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:33:30.591 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.587+0000 7f762d59a640 4 set_mon_vals no callback set 2024-03-25T18:33:30.592 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.588+0000 7f7626ffd640 1 -- 172.21.15.155:0/3816374487 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7f7618003c40 con 0x7f763013f710 2024-03-25T18:33:30.592 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.589+0000 7f762e59c640 10 monclient: get_auth_request con 0x7f761006dfb0 auth_method 0 2024-03-25T18:33:30.592 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.589+0000 7f7626ffd640 10 client.4623.objecter ms_handle_connect 0x7f761006dfb0 2024-03-25T18:33:30.599 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.596+0000 7f7626ffd640 1 -- 172.21.15.155:0/3816374487 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7f7618054900 con 0x7f763013f710 2024-03-25T18:33:30.599 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.596+0000 7f7626ffd640 10 client.4623.objecter ms_dispatch 0x7f7630000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:33:30.599 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.596+0000 7f7626ffd640 3 client.4623.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:33:30.599 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.596+0000 7f7626ffd640 3 client.4623.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:33:30.599 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.596+0000 7f7626ffd640 20 client.4623.objecter dump_active .. 0 homeless 2024-03-25T18:33:30.599 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.596+0000 7f7626ffd640 1 -- 172.21.15.155:0/3816374487 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7618050240 con 0x7f763013f710 2024-03-25T18:33:30.599 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.596+0000 7f7626ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:30.599 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.596+0000 7f7626ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:33:30.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.844+0000 7f7634ce4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:30.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.844+0000 7f7634ce4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:30.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.844+0000 7f7634ce4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:33:30.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.844+0000 7f7634ce4640 1 -- 172.21.15.155:0/3816374487 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f76300627b0 con 0x7f763013f710 2024-03-25T18:33:30.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.867+0000 7f7626ffd640 1 -- 172.21.15.155:0/3816374487 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7f7618052340 con 0x7f763013f710 2024-03-25T18:33:30.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.867+0000 7f7626ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:30.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.867+0000 7f7626ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:33:30.870 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:33:30.870 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:33:30.873 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.870+0000 7f7634ce4640 10 librados: watch_flush enter 2024-03-25T18:33:30.873 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.870+0000 7f7634ce4640 10 librados: watch_flush exit 2024-03-25T18:33:30.873 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.870+0000 7f7634ce4640 20 client.4623.objecter shutdown clearing up homeless session... 2024-03-25T18:33:30.873 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.870+0000 7f7634ce4640 10 client.4623.objecter successfully canceled tick 2024-03-25T18:33:30.873 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.870+0000 7f7634ce4640 1 -- 172.21.15.155:0/3816374487 >> v1:172.21.15.167:6800/2919070503 conn(0x7f761006dfb0 legacy=0x7f7610070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:30.873 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.870+0000 7f7634ce4640 10 monclient: shutdown 2024-03-25T18:33:30.874 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.870+0000 7f7634ce4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:30.874 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.870+0000 7f7634ce4640 1 -- 172.21.15.155:0/3816374487 >> v1:172.21.15.155:6789/0 conn(0x7f763013f710 legacy=0x7f7630139170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:30.874 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.870+0000 7f7634ce4640 1 -- 172.21.15.155:0/3816374487 shutdown_connections 2024-03-25T18:33:30.874 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.870+0000 7f7634ce4640 1 -- 172.21.15.155:0/3816374487 >> 172.21.15.155:0/3816374487 conn(0x7f76300b7ea0 msgr2=0x7f7630074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:30.874 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.870+0000 7f7634ce4640 1 -- 172.21.15.155:0/3816374487 shutdown_connections 2024-03-25T18:33:30.874 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.871+0000 7f7634ce4640 1 -- 172.21.15.155:0/3816374487 wait complete. 2024-03-25T18:33:30.874 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:30.871+0000 7f7634ce4640 1 librados: shutdown 2024-03-25T18:33:30.885 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:33:32.293 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:09:05.897 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:32.301 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:09:05.905 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:32.333 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:09:05.937 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:32.343 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:09:05.922 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:32.351 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:09:05.956 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:32.353 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:09:05.932 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:32.355 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:09:05.960 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:32.380 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:09:05.984 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:32.401 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:09:05.979 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:37.692 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:33:37.809 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.807+0000 7fa68f80d640 10 monclient: get_monmap_and_config 2024-03-25T18:33:37.809 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.807+0000 7fa68f80d640 10 monclient: build_initial_monmap 2024-03-25T18:33:37.809 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.807+0000 7fa68f80d640 10 monclient: monmap: 2024-03-25T18:33:37.809 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:33:37.809 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:33:37.809 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:33:37.808847+0000 2024-03-25T18:33:37.809 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:33:37.808847+0000 2024-03-25T18:33:37.809 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:37.809 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:33:37.810 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:33:37.810 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:33:37.810 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:33:37.810 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:33:37.811 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.808+0000 7fa68f80d640 1 Processor -- start 2024-03-25T18:33:37.811 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.808+0000 7fa68f80d640 1 -- start start 2024-03-25T18:33:37.811 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.808+0000 7fa68f80d640 10 monclient: init 2024-03-25T18:33:37.811 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.809+0000 7fa68f80d640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:37.811 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.809+0000 7fa68f80d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:33:37.811 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.809+0000 7fa68f80d640 10 monclient(hunting): picked mon.noname-c con 0x7fa6880a98f0 addr v1:172.21.15.167:6789/0 2024-03-25T18:33:37.811 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.809+0000 7fa68f80d640 10 monclient(hunting): picked mon.noname-b con 0x7fa68813f710 addr v1:172.21.15.155:6790/0 2024-03-25T18:33:37.811 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.810+0000 7fa68f80d640 10 monclient(hunting): picked mon.noname-a con 0x7fa6881430c0 addr v1:172.21.15.155:6789/0 2024-03-25T18:33:37.811 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.810+0000 7fa68f80d640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa688142520 con 0x7fa6881430c0 2024-03-25T18:33:37.811 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.810+0000 7fa68f80d640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6880b28a0 con 0x7fa6880a98f0 2024-03-25T18:33:37.811 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.810+0000 7fa68f80d640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6880b2a30 con 0x7fa68813f710 2024-03-25T18:33:37.811 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.810+0000 7fa68f80d640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:37.812 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.810+0000 7fa68f80d640 10 monclient(hunting): authenticate will time out at 8064.119892s 2024-03-25T18:33:37.820 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.819+0000 7fa68cda8640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fa68813f710 0x7fa6881429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:32896/0 (socket says 172.21.15.155:32896) 2024-03-25T18:33:37.820 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.819+0000 7fa68cda8640 1 -- 172.21.15.155:0/3716321640 learned_addr learned my addr 172.21.15.155:0/3716321640 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:33:37.829 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.828+0000 7fa67ffff640 1 -- 172.21.15.155:0/3716321640 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2698178104 0 0) 0x7fa6880b2a30 con 0x7fa68813f710 2024-03-25T18:33:37.829 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.828+0000 7fa67ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:33:37.829 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.828+0000 7fa67ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:37.829 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.828+0000 7fa67ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:37.829 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.828+0000 7fa67ffff640 10 monclient(hunting): my global_id is 4634 2024-03-25T18:33:37.829 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.828+0000 7fa67ffff640 1 -- 172.21.15.155:0/3716321640 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa674003520 con 0x7fa68813f710 2024-03-25T18:33:37.834 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.832+0000 7fa67ffff640 1 -- 172.21.15.155:0/3716321640 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2944180770 0 0) 0x7fa674003520 con 0x7fa68813f710 2024-03-25T18:33:37.834 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.832+0000 7fa67ffff640 1 -- 172.21.15.155:0/3716321640 >> v1:172.21.15.167:6789/0 conn(0x7fa6880a98f0 legacy=0x7fa6880a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:37.834 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.832+0000 7fa67ffff640 1 -- 172.21.15.155:0/3716321640 >> v1:172.21.15.155:6789/0 conn(0x7fa6881430c0 legacy=0x7fa68814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:37.834 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.832+0000 7fa67ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:33:37.834 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.832+0000 7fa67ffff640 1 monclient: found mon.noname-b 2024-03-25T18:33:37.834 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.832+0000 7fa67ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:37.834 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.832+0000 7fa67ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-25T18:33:37.834 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.832+0000 7fa67ffff640 1 -- 172.21.15.155:0/3716321640 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6880b44c0 con 0x7fa68813f710 2024-03-25T18:33:37.834 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.832+0000 7fa67ffff640 10 monclient: _finish_auth 0 2024-03-25T18:33:37.834 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.832+0000 7fa67ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:33:37.834 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.832+0000 7fa67ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:37.835 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.832+0000 7fa68f80d640 5 monclient: authenticate success, global_id 4634 2024-03-25T18:33:37.835 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.832+0000 7fa68f80d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:37.836 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.835+0000 7fa67ffff640 1 -- 172.21.15.155:0/3716321640 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fa678001690 con 0x7fa68813f710 2024-03-25T18:33:37.836 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.835+0000 7fa67ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:37.836 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.835+0000 7fa67ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:33:37.836 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.835+0000 7fa67ffff640 10 monclient: dump: 2024-03-25T18:33:37.836 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:33:37.836 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:33:37.837 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:37.837 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:37.837 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:37.837 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:33:37.837 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:33:37.837 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:33:37.837 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:33:37.837 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:33:37.837 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.835+0000 7fa68f80d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:37.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.852+0000 7fa67ffff640 1 -- 172.21.15.155:0/3716321640 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa678001920 con 0x7fa68813f710 2024-03-25T18:33:37.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.852+0000 7fa67ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:37.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.852+0000 7fa67ffff640 1 -- 172.21.15.155:0/3716321640 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fa678001c30 con 0x7fa68813f710 2024-03-25T18:33:37.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.852+0000 7fa67ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:37.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.852+0000 7fa67ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:33:37.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.852+0000 7fa67ffff640 10 monclient: dump: 2024-03-25T18:33:37.853 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:33:37.853 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:33:37.853 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:37.854 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:37.854 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:37.854 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:33:37.854 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:33:37.854 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:33:37.854 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:33:37.854 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:33:37.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.852+0000 7fa68f80d640 10 monclient: get_monmap_and_config success 2024-03-25T18:33:37.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.852+0000 7fa68f80d640 4 set_mon_vals no callback set 2024-03-25T18:33:37.856 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.853+0000 7fa68f80d640 10 monclient: shutdown 2024-03-25T18:33:37.856 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.853+0000 7fa68f80d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:37.856 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.853+0000 7fa68f80d640 1 -- 172.21.15.155:0/3716321640 >> v1:172.21.15.155:6790/0 conn(0x7fa68813f710 legacy=0x7fa6881429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:37.856 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.853+0000 7fa68f80d640 1 -- 172.21.15.155:0/3716321640 shutdown_connections 2024-03-25T18:33:37.856 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.853+0000 7fa68f80d640 1 -- 172.21.15.155:0/3716321640 >> 172.21.15.155:0/3716321640 conn(0x7fa6880b7ea0 msgr2=0x7fa6880b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:37.856 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.853+0000 7fa68f80d640 1 -- 172.21.15.155:0/3716321640 shutdown_connections 2024-03-25T18:33:37.857 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.853+0000 7fa68f80d640 1 -- 172.21.15.155:0/3716321640 wait complete. 2024-03-25T18:33:37.857 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.853+0000 7fa68f80d640 10 monclient: build_initial_monmap 2024-03-25T18:33:37.857 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.853+0000 7fa68f80d640 10 monclient: monmap: 2024-03-25T18:33:37.857 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:33:37.857 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:33:37.857 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:33:37.857 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:33:37.857 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:37.857 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:33:37.857 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:33:37.857 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:33:37.857 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:33:37.857 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:33:37.857 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.854+0000 7fa68f80d640 1 librados: starting msgr at 2024-03-25T18:33:37.857 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.854+0000 7fa68f80d640 1 librados: starting objecter 2024-03-25T18:33:37.857 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.854+0000 7fa68f80d640 1 Processor -- start 2024-03-25T18:33:37.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.854+0000 7fa68f80d640 1 -- start start 2024-03-25T18:33:37.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.854+0000 7fa68f80d640 1 librados: setting wanted keys 2024-03-25T18:33:37.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.854+0000 7fa68f80d640 1 librados: calling monclient init 2024-03-25T18:33:37.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.854+0000 7fa68f80d640 10 monclient: init 2024-03-25T18:33:37.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.854+0000 7fa68f80d640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:37.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.854+0000 7fa68f80d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:33:37.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.854+0000 7fa68f80d640 10 monclient(hunting): picked mon.noname-a con 0x7fa6880a98f0 addr v1:172.21.15.155:6789/0 2024-03-25T18:33:37.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.854+0000 7fa68f80d640 10 monclient(hunting): picked mon.noname-c con 0x7fa68813f710 addr v1:172.21.15.155:6790/0 2024-03-25T18:33:37.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.854+0000 7fa68f80d640 10 monclient(hunting): picked mon.noname-b con 0x7fa6881430c0 addr v1:172.21.15.167:6789/0 2024-03-25T18:33:37.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.854+0000 7fa68f80d640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6880abb30 con 0x7fa6880a98f0 2024-03-25T18:33:37.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.854+0000 7fa68f80d640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6881d8390 con 0x7fa6881430c0 2024-03-25T18:33:37.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.854+0000 7fa68f80d640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa688137f50 con 0x7fa68813f710 2024-03-25T18:33:37.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.854+0000 7fa68f80d640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:37.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.854+0000 7fa68f80d640 10 monclient(hunting): authenticate will time out at 8064.164664s 2024-03-25T18:33:37.861 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.860+0000 7fa68ddaa640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7fa6881430c0 0x7fa688134010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:43134/0 (socket says 172.21.15.155:43134) 2024-03-25T18:33:37.862 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.860+0000 7fa68ddaa640 1 -- 172.21.15.155:0/1800653125 learned_addr learned my addr 172.21.15.155:0/1800653125 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:33:37.862 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.861+0000 7fa67dffb640 10 client.?.objecter ms_handle_connect 0x7fa6881430c0 2024-03-25T18:33:37.862 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.861+0000 7fa67dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:37.872 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.871+0000 7fa67dffb640 10 client.?.objecter ms_handle_connect 0x7fa6880a98f0 2024-03-25T18:33:37.872 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.871+0000 7fa67dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:37.882 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.881+0000 7fa67dffb640 1 -- 172.21.15.155:0/1800653125 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2292051971 0 0) 0x7fa6880abb30 con 0x7fa6880a98f0 2024-03-25T18:33:37.882 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.881+0000 7fa67dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:33:37.882 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.881+0000 7fa67dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:37.883 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.881+0000 7fa67dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:37.883 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.881+0000 7fa67dffb640 10 monclient(hunting): my global_id is 4629 2024-03-25T18:33:37.883 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.881+0000 7fa67dffb640 1 -- 172.21.15.155:0/1800653125 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa66c004eb0 con 0x7fa6880a98f0 2024-03-25T18:33:37.883 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.882+0000 7fa67dffb640 10 client.?.objecter ms_handle_connect 0x7fa68813f710 2024-03-25T18:33:37.883 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.882+0000 7fa67dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:37.887 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.886+0000 7fa67dffb640 1 -- 172.21.15.155:0/1800653125 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1099290469 0 0) 0x7fa6881d8390 con 0x7fa6881430c0 2024-03-25T18:33:37.887 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.886+0000 7fa67dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:33:37.887 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.886+0000 7fa67dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:37.887 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.886+0000 7fa67dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:37.887 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.886+0000 7fa67dffb640 10 monclient(hunting): my global_id is 4624 2024-03-25T18:33:37.887 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.886+0000 7fa67dffb640 1 -- 172.21.15.155:0/1800653125 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6880abb30 con 0x7fa6881430c0 2024-03-25T18:33:37.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.893+0000 7fa67dffb640 1 -- 172.21.15.155:0/1800653125 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2113783212 0 0) 0x7fa688137f50 con 0x7fa68813f710 2024-03-25T18:33:37.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.893+0000 7fa67dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:33:37.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.893+0000 7fa67dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:37.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.893+0000 7fa67dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:37.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.893+0000 7fa67dffb640 10 monclient(hunting): my global_id is 4637 2024-03-25T18:33:37.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.893+0000 7fa67dffb640 1 -- 172.21.15.155:0/1800653125 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6881d8390 con 0x7fa68813f710 2024-03-25T18:33:37.900 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.899+0000 7fa67dffb640 1 -- 172.21.15.155:0/1800653125 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3377376103 0 0) 0x7fa66c004eb0 con 0x7fa6880a98f0 2024-03-25T18:33:37.900 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.899+0000 7fa67dffb640 1 -- 172.21.15.155:0/1800653125 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa688137f50 con 0x7fa6880a98f0 2024-03-25T18:33:37.900 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.899+0000 7fa67dffb640 1 -- 172.21.15.155:0/1800653125 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fa6840018c0 con 0x7fa6880a98f0 2024-03-25T18:33:37.900 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.899+0000 7fa67dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:37.900 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.899+0000 7fa67dffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:33:37.900 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.899+0000 7fa67dffb640 10 monclient(hunting): dump: 2024-03-25T18:33:37.901 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:33:37.901 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:33:37.901 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:37.901 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:37.901 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:37.901 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:33:37.901 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:33:37.901 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:33:37.901 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:33:37.901 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:33:37.901 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.899+0000 7fa67dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:33:37.907 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.905+0000 7fa67dffb640 1 -- 172.21.15.155:0/1800653125 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 735086500 0 0) 0x7fa6881d8390 con 0x7fa68813f710 2024-03-25T18:33:37.907 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.905+0000 7fa67dffb640 1 -- 172.21.15.155:0/1800653125 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa66c004eb0 con 0x7fa68813f710 2024-03-25T18:33:37.907 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.906+0000 7fa67dffb640 1 -- 172.21.15.155:0/1800653125 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fa6780014b0 con 0x7fa68813f710 2024-03-25T18:33:37.907 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.906+0000 7fa67dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:37.907 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.906+0000 7fa67dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:33:37.907 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.906+0000 7fa67dffb640 10 monclient(hunting): dump: 2024-03-25T18:33:37.907 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:33:37.907 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:33:37.907 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:37.907 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:37.907 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:37.907 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:33:37.908 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:33:37.908 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:33:37.908 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:33:37.908 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:33:37.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.914+0000 7fa67dffb640 1 -- 172.21.15.155:0/1800653125 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3619496563 0 0) 0x7fa688137f50 con 0x7fa6880a98f0 2024-03-25T18:33:37.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.914+0000 7fa67dffb640 1 -- 172.21.15.155:0/1800653125 >> v1:172.21.15.155:6790/0 conn(0x7fa68813f710 legacy=0x7fa6881d7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:37.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.914+0000 7fa67dffb640 1 -- 172.21.15.155:0/1800653125 >> v1:172.21.15.167:6789/0 conn(0x7fa6881430c0 legacy=0x7fa688134010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:37.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.915+0000 7fa67dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:33:37.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.915+0000 7fa67dffb640 1 monclient: found mon.a 2024-03-25T18:33:37.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.915+0000 7fa67dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:37.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.915+0000 7fa67dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:33:37.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.915+0000 7fa67dffb640 1 -- 172.21.15.155:0/1800653125 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa688134720 con 0x7fa6880a98f0 2024-03-25T18:33:37.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.915+0000 7fa67dffb640 10 monclient: _finish_auth 0 2024-03-25T18:33:37.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.915+0000 7fa67dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:33:37.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.915+0000 7fa67dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:37.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.915+0000 7fa68f80d640 5 monclient: authenticate success, global_id 4629 2024-03-25T18:33:37.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.915+0000 7fa68f80d640 10 monclient: _renew_subs 2024-03-25T18:33:37.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.915+0000 7fa68f80d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:33:37.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.915+0000 7fa68f80d640 1 -- 172.21.15.155:0/1800653125 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa688134910 con 0x7fa6880a98f0 2024-03-25T18:33:37.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.915+0000 7fa68f80d640 10 client.4629.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:33:37.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.915+0000 7fa68f80d640 10 monclient: _renew_subs 2024-03-25T18:33:37.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.915+0000 7fa68f80d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:33:37.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.915+0000 7fa68f80d640 1 -- 172.21.15.155:0/1800653125 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa688135050 con 0x7fa6880a98f0 2024-03-25T18:33:37.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.915+0000 7fa68f80d640 1 librados: init done 2024-03-25T18:33:37.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.916+0000 7fa68f80d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:37.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.916+0000 7fa68f80d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:37.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.916+0000 7fa68f80d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:33:37.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:37.916+0000 7fa68f80d640 1 -- 172.21.15.155:0/1800653125 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6880627b0 con 0x7fa6880a98f0 2024-03-25T18:33:38.218 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:38.216+0000 7fa67dffb640 1 -- 172.21.15.155:0/1800653125 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa6840035f0 con 0x7fa6880a98f0 2024-03-25T18:33:38.218 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:38.216+0000 7fa67dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:38.218 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:38.216+0000 7fa67dffb640 1 -- 172.21.15.155:0/1800653125 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fa684003980 con 0x7fa6880a98f0 2024-03-25T18:33:38.218 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:38.216+0000 7fa67dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:38.218 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:38.216+0000 7fa67dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:33:38.218 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:38.216+0000 7fa67dffb640 10 monclient: dump: 2024-03-25T18:33:38.218 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:33:38.218 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:33:38.218 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:38.218 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:38.219 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:38.219 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:33:38.219 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:33:38.219 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:33:38.219 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:33:38.219 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:33:38.219 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:38.216+0000 7fa67f7fe640 4 set_mon_vals no callback set 2024-03-25T18:33:38.220 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:38.217+0000 7fa67dffb640 1 -- 172.21.15.155:0/1800653125 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7fa684003ba0 con 0x7fa6880a98f0 2024-03-25T18:33:38.220 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:38.218+0000 7fa68cda8640 10 monclient: get_auth_request con 0x7fa66c06df70 auth_method 0 2024-03-25T18:33:38.220 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:38.218+0000 7fa67dffb640 10 client.4629.objecter ms_handle_connect 0x7fa66c06df70 2024-03-25T18:33:38.227 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:38.225+0000 7fa67dffb640 1 -- 172.21.15.155:0/1800653125 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7fa684053520 con 0x7fa6880a98f0 2024-03-25T18:33:38.227 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:38.225+0000 7fa67dffb640 10 client.4629.objecter ms_dispatch 0x7fa688000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:33:38.227 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:38.225+0000 7fa67dffb640 3 client.4629.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:33:38.227 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:38.226+0000 7fa67dffb640 3 client.4629.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:33:38.227 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:38.226+0000 7fa67dffb640 20 client.4629.objecter dump_active .. 0 homeless 2024-03-25T18:33:38.227 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:38.226+0000 7fa67dffb640 1 -- 172.21.15.155:0/1800653125 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa68404f030 con 0x7fa6880a98f0 2024-03-25T18:33:38.227 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:38.226+0000 7fa67dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:38.227 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:38.226+0000 7fa67dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:33:38.478 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:38.477+0000 7fa68f80d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:38.478 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:38.477+0000 7fa68f80d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:38.479 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:38.477+0000 7fa68f80d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:33:38.479 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:38.477+0000 7fa68f80d640 1 -- 172.21.15.155:0/1800653125 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa68813fdc0 con 0x7fa6880a98f0 2024-03-25T18:33:38.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:38.494+0000 7fa67dffb640 1 -- 172.21.15.155:0/1800653125 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7fa684050f60 con 0x7fa6880a98f0 2024-03-25T18:33:38.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:38.495+0000 7fa67dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:38.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:38.495+0000 7fa67dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:33:38.496 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:33:38.496 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:33:38.499 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:38.497+0000 7fa68f80d640 10 librados: watch_flush enter 2024-03-25T18:33:38.499 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:38.497+0000 7fa68f80d640 10 librados: watch_flush exit 2024-03-25T18:33:38.499 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:38.497+0000 7fa68f80d640 20 client.4629.objecter shutdown clearing up homeless session... 2024-03-25T18:33:38.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:38.497+0000 7fa68f80d640 10 client.4629.objecter successfully canceled tick 2024-03-25T18:33:38.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:38.498+0000 7fa68f80d640 1 -- 172.21.15.155:0/1800653125 >> v1:172.21.15.167:6800/2919070503 conn(0x7fa66c06df70 legacy=0x7fa66c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:38.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:38.498+0000 7fa68f80d640 10 monclient: shutdown 2024-03-25T18:33:38.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:38.498+0000 7fa68f80d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:38.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:38.498+0000 7fa68f80d640 1 -- 172.21.15.155:0/1800653125 >> v1:172.21.15.155:6789/0 conn(0x7fa6880a98f0 legacy=0x7fa688138500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:38.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:38.498+0000 7fa68f80d640 1 -- 172.21.15.155:0/1800653125 shutdown_connections 2024-03-25T18:33:38.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:38.498+0000 7fa68f80d640 1 -- 172.21.15.155:0/1800653125 >> 172.21.15.155:0/1800653125 conn(0x7fa6880b7ea0 msgr2=0x7fa688074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:38.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:38.498+0000 7fa68f80d640 1 -- 172.21.15.155:0/1800653125 shutdown_connections 2024-03-25T18:33:38.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:38.498+0000 7fa68f80d640 1 -- 172.21.15.155:0/1800653125 wait complete. 2024-03-25T18:33:38.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:38.498+0000 7fa68f80d640 1 librados: shutdown 2024-03-25T18:33:38.513 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:33:45.320 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:33:45.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.431+0000 7fe6b37e8640 10 monclient: get_monmap_and_config 2024-03-25T18:33:45.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.431+0000 7fe6b37e8640 10 monclient: build_initial_monmap 2024-03-25T18:33:45.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.431+0000 7fe6b37e8640 10 monclient: monmap: 2024-03-25T18:33:45.433 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:33:45.433 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:33:45.433 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:33:45.432687+0000 2024-03-25T18:33:45.433 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:33:45.432687+0000 2024-03-25T18:33:45.433 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:45.433 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:33:45.434 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:33:45.434 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:33:45.434 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:33:45.434 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:33:45.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.432+0000 7fe6b37e8640 1 Processor -- start 2024-03-25T18:33:45.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.432+0000 7fe6b37e8640 1 -- start start 2024-03-25T18:33:45.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.432+0000 7fe6b37e8640 10 monclient: init 2024-03-25T18:33:45.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.433+0000 7fe6b37e8640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:45.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.433+0000 7fe6b37e8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:33:45.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.433+0000 7fe6b37e8640 10 monclient(hunting): picked mon.noname-c con 0x7fe6ac0b4d60 addr v1:172.21.15.167:6789/0 2024-03-25T18:33:45.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.433+0000 7fe6b37e8640 10 monclient(hunting): picked mon.noname-b con 0x7fe6ac0b1730 addr v1:172.21.15.155:6790/0 2024-03-25T18:33:45.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.434+0000 7fe6b37e8640 10 monclient(hunting): picked mon.noname-a con 0x7fe6ac14be90 addr v1:172.21.15.155:6789/0 2024-03-25T18:33:45.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.434+0000 7fe6b37e8640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6ac0af940 con 0x7fe6ac14be90 2024-03-25T18:33:45.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.434+0000 7fe6b37e8640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6ac0b0a70 con 0x7fe6ac0b4d60 2024-03-25T18:33:45.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.434+0000 7fe6b37e8640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6ac0b0d50 con 0x7fe6ac0b1730 2024-03-25T18:33:45.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.434+0000 7fe6b37e8640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:45.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.434+0000 7fe6b37e8640 10 monclient(hunting): authenticate will time out at 8071.743834s 2024-03-25T18:33:45.451 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.450+0000 7fe6b0d83640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fe6ac0b1730 0x7fe6ac0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:37580/0 (socket says 172.21.15.155:37580) 2024-03-25T18:33:45.451 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.450+0000 7fe6b0d83640 1 -- 172.21.15.155:0/2389728526 learned_addr learned my addr 172.21.15.155:0/2389728526 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:33:45.463 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.462+0000 7fe6a3fff640 1 -- 172.21.15.155:0/2389728526 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1241831106 0 0) 0x7fe6ac0b0d50 con 0x7fe6ac0b1730 2024-03-25T18:33:45.464 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.462+0000 7fe6a3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:33:45.464 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.462+0000 7fe6a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:45.464 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.462+0000 7fe6a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:45.464 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.462+0000 7fe6a3fff640 10 monclient(hunting): my global_id is 4640 2024-03-25T18:33:45.464 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.462+0000 7fe6a3fff640 1 -- 172.21.15.155:0/2389728526 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe698003520 con 0x7fe6ac0b1730 2024-03-25T18:33:45.471 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.469+0000 7fe6a3fff640 1 -- 172.21.15.155:0/2389728526 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1928137478 0 0) 0x7fe698003520 con 0x7fe6ac0b1730 2024-03-25T18:33:45.471 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.469+0000 7fe6a3fff640 1 -- 172.21.15.155:0/2389728526 >> v1:172.21.15.167:6789/0 conn(0x7fe6ac0b4d60 legacy=0x7fe6ac13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:45.471 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.469+0000 7fe6a3fff640 1 -- 172.21.15.155:0/2389728526 >> v1:172.21.15.155:6789/0 conn(0x7fe6ac14be90 legacy=0x7fe6ac14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:45.471 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.469+0000 7fe6a3fff640 10 monclient: _finish_hunting 0 2024-03-25T18:33:45.471 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.469+0000 7fe6a3fff640 1 monclient: found mon.noname-b 2024-03-25T18:33:45.471 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.469+0000 7fe6a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:45.471 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.469+0000 7fe6a3fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-25T18:33:45.471 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.469+0000 7fe6a3fff640 1 -- 172.21.15.155:0/2389728526 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6ac0b1030 con 0x7fe6ac0b1730 2024-03-25T18:33:45.472 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.469+0000 7fe6a3fff640 10 monclient: _finish_auth 0 2024-03-25T18:33:45.472 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.469+0000 7fe6a3fff640 10 monclient: _check_auth_tickets 2024-03-25T18:33:45.472 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.469+0000 7fe6a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:45.472 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.469+0000 7fe6a3fff640 1 -- 172.21.15.155:0/2389728526 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fe69c001690 con 0x7fe6ac0b1730 2024-03-25T18:33:45.472 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.469+0000 7fe6b37e8640 5 monclient: authenticate success, global_id 4640 2024-03-25T18:33:45.472 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.469+0000 7fe6b37e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:45.472 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.469+0000 7fe6a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:45.472 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.469+0000 7fe6a3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:33:45.472 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.469+0000 7fe6a3fff640 10 monclient: dump: 2024-03-25T18:33:45.472 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:33:45.472 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:33:45.472 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:45.472 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:45.473 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:45.473 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:33:45.473 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:33:45.473 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:33:45.473 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:33:45.473 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:33:45.473 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.469+0000 7fe6b37e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:45.473 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.472+0000 7fe6a3fff640 1 -- 172.21.15.155:0/2389728526 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe69c0017f0 con 0x7fe6ac0b1730 2024-03-25T18:33:45.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.472+0000 7fe6a3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:45.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.472+0000 7fe6b37e8640 10 monclient: get_monmap_and_config success 2024-03-25T18:33:45.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.472+0000 7fe6b37e8640 4 set_mon_vals no callback set 2024-03-25T18:33:45.475 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.473+0000 7fe6b37e8640 10 monclient: shutdown 2024-03-25T18:33:45.475 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.473+0000 7fe6b37e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:45.475 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.473+0000 7fe6b37e8640 1 -- 172.21.15.155:0/2389728526 >> v1:172.21.15.155:6790/0 conn(0x7fe6ac0b1730 legacy=0x7fe6ac0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:45.475 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.473+0000 7fe6b37e8640 1 -- 172.21.15.155:0/2389728526 shutdown_connections 2024-03-25T18:33:45.475 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.473+0000 7fe6b37e8640 1 -- 172.21.15.155:0/2389728526 >> 172.21.15.155:0/2389728526 conn(0x7fe6ac0617a0 msgr2=0x7fe6ac061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:45.475 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.473+0000 7fe6b37e8640 1 -- 172.21.15.155:0/2389728526 shutdown_connections 2024-03-25T18:33:45.475 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.473+0000 7fe6b37e8640 1 -- 172.21.15.155:0/2389728526 wait complete. 2024-03-25T18:33:45.477 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.474+0000 7fe6b37e8640 10 monclient: build_initial_monmap 2024-03-25T18:33:45.477 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.474+0000 7fe6b37e8640 10 monclient: monmap: 2024-03-25T18:33:45.477 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:33:45.477 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:33:45.477 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:33:45.477 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:33:45.477 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:45.477 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:33:45.478 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:33:45.478 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:33:45.478 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:33:45.478 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:33:45.478 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.474+0000 7fe6b37e8640 1 librados: starting msgr at 2024-03-25T18:33:45.478 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.474+0000 7fe6b37e8640 1 librados: starting objecter 2024-03-25T18:33:45.478 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.474+0000 7fe6b37e8640 1 Processor -- start 2024-03-25T18:33:45.478 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.475+0000 7fe6b37e8640 1 -- start start 2024-03-25T18:33:45.478 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.475+0000 7fe6b37e8640 1 librados: setting wanted keys 2024-03-25T18:33:45.478 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.475+0000 7fe6b37e8640 1 librados: calling monclient init 2024-03-25T18:33:45.479 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.475+0000 7fe6b37e8640 10 monclient: init 2024-03-25T18:33:45.479 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.475+0000 7fe6b37e8640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:45.479 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.475+0000 7fe6b37e8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:33:45.479 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.475+0000 7fe6b37e8640 10 monclient(hunting): picked mon.noname-c con 0x7fe6ac0b1730 addr v1:172.21.15.155:6790/0 2024-03-25T18:33:45.479 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.475+0000 7fe6b37e8640 10 monclient(hunting): picked mon.noname-a con 0x7fe6ac0b4d60 addr v1:172.21.15.155:6789/0 2024-03-25T18:33:45.479 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.475+0000 7fe6b37e8640 10 monclient(hunting): picked mon.noname-b con 0x7fe6ac14be90 addr v1:172.21.15.167:6789/0 2024-03-25T18:33:45.479 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.475+0000 7fe6b37e8640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6ac1daf00 con 0x7fe6ac0b4d60 2024-03-25T18:33:45.479 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.475+0000 7fe6b37e8640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6ac138820 con 0x7fe6ac14be90 2024-03-25T18:33:45.479 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.475+0000 7fe6b37e8640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6ac1dab40 con 0x7fe6ac0b1730 2024-03-25T18:33:45.479 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.475+0000 7fe6b37e8640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:45.479 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.475+0000 7fe6b37e8640 10 monclient(hunting): authenticate will time out at 8071.785290s 2024-03-25T18:33:45.487 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.486+0000 7fe6b0d83640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fe6ac0b4d60 0x7fe6ac138110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:35586/0 (socket says 172.21.15.155:35586) 2024-03-25T18:33:45.487 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.486+0000 7fe6b0d83640 1 -- 172.21.15.155:0/1149404756 learned_addr learned my addr 172.21.15.155:0/1149404756 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:33:45.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.488+0000 7fe6a1ffb640 10 client.?.objecter ms_handle_connect 0x7fe6ac0b4d60 2024-03-25T18:33:45.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.488+0000 7fe6a1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:45.494 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.493+0000 7fe6a1ffb640 10 client.?.objecter ms_handle_connect 0x7fe6ac14be90 2024-03-25T18:33:45.494 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.493+0000 7fe6a1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:45.499 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.498+0000 7fe6a1ffb640 10 client.?.objecter ms_handle_connect 0x7fe6ac0b1730 2024-03-25T18:33:45.499 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.498+0000 7fe6a1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:45.505 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.503+0000 7fe6a1ffb640 1 -- 172.21.15.155:0/1149404756 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2359341546 0 0) 0x7fe6ac138820 con 0x7fe6ac14be90 2024-03-25T18:33:45.505 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.504+0000 7fe6a1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:33:45.505 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.504+0000 7fe6a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:45.505 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.504+0000 7fe6a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:45.505 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.504+0000 7fe6a1ffb640 10 monclient(hunting): my global_id is 4630 2024-03-25T18:33:45.505 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.504+0000 7fe6a1ffb640 1 -- 172.21.15.155:0/1149404756 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe690004eb0 con 0x7fe6ac14be90 2024-03-25T18:33:45.506 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.505+0000 7fe6a1ffb640 1 -- 172.21.15.155:0/1149404756 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3006041701 0 0) 0x7fe6ac1daf00 con 0x7fe6ac0b4d60 2024-03-25T18:33:45.506 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.505+0000 7fe6a1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:33:45.506 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.505+0000 7fe6a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:45.506 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.505+0000 7fe6a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:45.506 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.505+0000 7fe6a1ffb640 10 monclient(hunting): my global_id is 4635 2024-03-25T18:33:45.506 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.505+0000 7fe6a1ffb640 1 -- 172.21.15.155:0/1149404756 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6ac138820 con 0x7fe6ac0b4d60 2024-03-25T18:33:45.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.512+0000 7fe6a1ffb640 1 -- 172.21.15.155:0/1149404756 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2111018077 0 0) 0x7fe6ac1dab40 con 0x7fe6ac0b1730 2024-03-25T18:33:45.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.512+0000 7fe6a1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:33:45.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.512+0000 7fe6a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:45.514 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.512+0000 7fe6a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:45.514 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.512+0000 7fe6a1ffb640 10 monclient(hunting): my global_id is 4643 2024-03-25T18:33:45.514 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.512+0000 7fe6a1ffb640 1 -- 172.21.15.155:0/1149404756 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6ac1daf00 con 0x7fe6ac0b1730 2024-03-25T18:33:45.537 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.536+0000 7fe6a1ffb640 1 -- 172.21.15.155:0/1149404756 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 218052170 0 0) 0x7fe6ac138820 con 0x7fe6ac0b4d60 2024-03-25T18:33:45.537 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.536+0000 7fe6a1ffb640 1 -- 172.21.15.155:0/1149404756 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe6ac1dab40 con 0x7fe6ac0b4d60 2024-03-25T18:33:45.537 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.536+0000 7fe6a1ffb640 1 -- 172.21.15.155:0/1149404756 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fe69c001ce0 con 0x7fe6ac0b4d60 2024-03-25T18:33:45.538 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.536+0000 7fe6a1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:45.538 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.536+0000 7fe6a1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:33:45.538 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.536+0000 7fe6a1ffb640 10 monclient(hunting): dump: 2024-03-25T18:33:45.538 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:33:45.538 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:33:45.538 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:45.538 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:45.538 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:45.538 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:33:45.538 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:33:45.538 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:33:45.539 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:33:45.539 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:33:45.539 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.536+0000 7fe6a1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:33:45.539 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.538+0000 7fe6a1ffb640 1 -- 172.21.15.155:0/1149404756 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1187499215 0 0) 0x7fe6ac1daf00 con 0x7fe6ac0b1730 2024-03-25T18:33:45.540 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.538+0000 7fe6a1ffb640 1 -- 172.21.15.155:0/1149404756 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe6ac138820 con 0x7fe6ac0b1730 2024-03-25T18:33:45.540 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.538+0000 7fe6a1ffb640 1 -- 172.21.15.155:0/1149404756 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fe6a8001860 con 0x7fe6ac0b1730 2024-03-25T18:33:45.540 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.538+0000 7fe6a1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:45.540 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.538+0000 7fe6a1ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:33:45.540 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.538+0000 7fe6a1ffb640 10 monclient(hunting): dump: 2024-03-25T18:33:45.540 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:33:45.540 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:33:45.540 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:45.540 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:45.540 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:45.540 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:33:45.540 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:33:45.540 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:33:45.541 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:33:45.541 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:33:45.542 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.540+0000 7fe6a1ffb640 1 -- 172.21.15.155:0/1149404756 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2908053746 0 0) 0x7fe690004eb0 con 0x7fe6ac14be90 2024-03-25T18:33:45.542 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.540+0000 7fe6a1ffb640 1 -- 172.21.15.155:0/1149404756 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe6ac1daf00 con 0x7fe6ac14be90 2024-03-25T18:33:45.542 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.540+0000 7fe6a1ffb640 1 -- 172.21.15.155:0/1149404756 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fe6a4001b90 con 0x7fe6ac14be90 2024-03-25T18:33:45.542 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.540+0000 7fe6a1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:45.542 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.540+0000 7fe6a1ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:33:45.542 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.540+0000 7fe6a1ffb640 10 monclient(hunting): dump: 2024-03-25T18:33:45.542 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:33:45.542 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:33:45.543 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:45.543 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:45.543 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:45.543 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:33:45.543 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:33:45.543 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:33:45.543 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:33:45.543 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:33:45.555 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.553+0000 7fe6a1ffb640 1 -- 172.21.15.155:0/1149404756 <== mon.1 v1:172.21.15.167:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1753386945 0 0) 0x7fe6ac1daf00 con 0x7fe6ac14be90 2024-03-25T18:33:45.555 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.553+0000 7fe6a1ffb640 1 -- 172.21.15.155:0/1149404756 >> v1:172.21.15.155:6790/0 conn(0x7fe6ac0b1730 legacy=0x7fe6ac1db420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:45.555 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.553+0000 7fe6a1ffb640 1 -- 172.21.15.155:0/1149404756 >> v1:172.21.15.155:6789/0 conn(0x7fe6ac0b4d60 legacy=0x7fe6ac138110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:45.555 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.553+0000 7fe6a1ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:33:45.555 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.553+0000 7fe6a1ffb640 1 monclient: found mon.b 2024-03-25T18:33:45.555 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.553+0000 7fe6a1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:45.555 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.553+0000 7fe6a1ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:33:45.556 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.553+0000 7fe6a1ffb640 1 -- 172.21.15.155:0/1149404756 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6ac1dacd0 con 0x7fe6ac14be90 2024-03-25T18:33:45.556 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.553+0000 7fe6a1ffb640 10 monclient: _finish_auth 0 2024-03-25T18:33:45.556 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.553+0000 7fe6a1ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:33:45.556 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.553+0000 7fe6a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:45.556 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.553+0000 7fe6b37e8640 5 monclient: authenticate success, global_id 4630 2024-03-25T18:33:45.556 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.553+0000 7fe6b37e8640 10 monclient: _renew_subs 2024-03-25T18:33:45.556 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.553+0000 7fe6b37e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:33:45.556 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.553+0000 7fe6b37e8640 1 -- 172.21.15.155:0/1149404756 --> v1:172.21.15.167:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6ac142bb0 con 0x7fe6ac14be90 2024-03-25T18:33:45.556 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.553+0000 7fe6b37e8640 10 client.4630.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:33:45.556 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.553+0000 7fe6b37e8640 10 monclient: _renew_subs 2024-03-25T18:33:45.556 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.553+0000 7fe6b37e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:33:45.556 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.553+0000 7fe6b37e8640 1 -- 172.21.15.155:0/1149404756 --> v1:172.21.15.167:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6ac142940 con 0x7fe6ac14be90 2024-03-25T18:33:45.556 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.553+0000 7fe6b37e8640 1 librados: init done 2024-03-25T18:33:45.557 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.554+0000 7fe6b37e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:45.557 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.554+0000 7fe6b37e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:45.557 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.554+0000 7fe6b37e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:33:45.557 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.554+0000 7fe6b37e8640 1 -- 172.21.15.155:0/1149404756 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6ac0627b0 con 0x7fe6ac14be90 2024-03-25T18:33:45.839 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.837+0000 7fe6a1ffb640 1 -- 172.21.15.155:0/1149404756 <== mon.1 v1:172.21.15.167:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe6a4002060 con 0x7fe6ac14be90 2024-03-25T18:33:45.839 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.837+0000 7fe6a1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:45.839 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.837+0000 7fe6a1ffb640 1 -- 172.21.15.155:0/1149404756 <== mon.1 v1:172.21.15.167:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fe6a4003b10 con 0x7fe6ac14be90 2024-03-25T18:33:45.839 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.837+0000 7fe6a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:45.839 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.837+0000 7fe6a1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:33:45.840 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.838+0000 7fe6a1ffb640 10 monclient: dump: 2024-03-25T18:33:45.840 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:33:45.840 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:33:45.840 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:45.840 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:45.840 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:45.840 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:33:45.840 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:33:45.840 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:33:45.840 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:33:45.840 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:33:45.840 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.837+0000 7fe6a37fe640 4 set_mon_vals no callback set 2024-03-25T18:33:45.841 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.838+0000 7fe6a1ffb640 1 -- 172.21.15.155:0/1149404756 <== mon.1 v1:172.21.15.167:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7fe6a4003d30 con 0x7fe6ac14be90 2024-03-25T18:33:45.841 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.839+0000 7fe6b1584640 10 monclient: get_auth_request con 0x7fe69006e0e0 auth_method 0 2024-03-25T18:33:45.841 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.839+0000 7fe6a1ffb640 10 client.4630.objecter ms_handle_connect 0x7fe69006e0e0 2024-03-25T18:33:45.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.844+0000 7fe6a1ffb640 1 -- 172.21.15.155:0/1149404756 <== mon.1 v1:172.21.15.167:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7fe6a40537e0 con 0x7fe6ac14be90 2024-03-25T18:33:45.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.844+0000 7fe6a1ffb640 10 client.4630.objecter ms_dispatch 0x7fe6ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:33:45.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.844+0000 7fe6a1ffb640 3 client.4630.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:33:45.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.844+0000 7fe6a1ffb640 3 client.4630.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:33:45.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.844+0000 7fe6a1ffb640 20 client.4630.objecter dump_active .. 0 homeless 2024-03-25T18:33:45.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.846+0000 7fe6a1ffb640 1 -- 172.21.15.155:0/1149404756 <== mon.1 v1:172.21.15.167:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe6a404f2f0 con 0x7fe6ac14be90 2024-03-25T18:33:45.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.846+0000 7fe6a1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:45.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:45.846+0000 7fe6a1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:33:46.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:46.093+0000 7fe6b37e8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:46.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:46.093+0000 7fe6b37e8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:46.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:46.093+0000 7fe6b37e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:33:46.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:46.093+0000 7fe6b37e8640 1 -- 172.21.15.155:0/1149404756 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe6ac14bc70 con 0x7fe6ac14be90 2024-03-25T18:33:46.126 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:46.125+0000 7fe6a1ffb640 1 -- 172.21.15.155:0/1149404756 <== mon.1 v1:172.21.15.167:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7fe6a4051220 con 0x7fe6ac14be90 2024-03-25T18:33:46.126 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:46.125+0000 7fe6a1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:46.126 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:46.125+0000 7fe6a1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:33:46.126 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:33:46.126 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:33:46.129 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:46.127+0000 7fe6b37e8640 10 librados: watch_flush enter 2024-03-25T18:33:46.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:46.127+0000 7fe6b37e8640 10 librados: watch_flush exit 2024-03-25T18:33:46.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:46.127+0000 7fe6b37e8640 20 client.4630.objecter shutdown clearing up homeless session... 2024-03-25T18:33:46.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:46.127+0000 7fe6b37e8640 10 client.4630.objecter successfully canceled tick 2024-03-25T18:33:46.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:46.128+0000 7fe6b37e8640 1 -- 172.21.15.155:0/1149404756 >> v1:172.21.15.167:6800/2919070503 conn(0x7fe69006e0e0 legacy=0x7fe6900705a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:46.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:46.128+0000 7fe6b37e8640 10 monclient: shutdown 2024-03-25T18:33:46.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:46.128+0000 7fe6b37e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:46.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:46.128+0000 7fe6b37e8640 1 -- 172.21.15.155:0/1149404756 >> v1:172.21.15.167:6789/0 conn(0x7fe6ac14be90 legacy=0x7fe6ac1e2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:46.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:46.128+0000 7fe6b37e8640 1 -- 172.21.15.155:0/1149404756 shutdown_connections 2024-03-25T18:33:46.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:46.128+0000 7fe6b37e8640 1 -- 172.21.15.155:0/1149404756 >> 172.21.15.155:0/1149404756 conn(0x7fe6ac0617a0 msgr2=0x7fe6ac0b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:46.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:46.128+0000 7fe6b37e8640 1 -- 172.21.15.155:0/1149404756 shutdown_connections 2024-03-25T18:33:46.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:46.128+0000 7fe6b37e8640 1 -- 172.21.15.155:0/1149404756 wait complete. 2024-03-25T18:33:46.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:46.128+0000 7fe6b37e8640 1 librados: shutdown 2024-03-25T18:33:46.141 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:33:52.390 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:09:25.994 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:52.395 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:09:25.999 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:52.428 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:09:26.033 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:52.450 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:09:26.029 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:52.453 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:09:26.057 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:52.455 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:09:26.033 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:52.460 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:09:26.064 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:52.481 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:09:26.060 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:52.482 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:09:26.086 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:52.946 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:33:53.059 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.057+0000 7fc0b8aa8640 10 monclient: get_monmap_and_config 2024-03-25T18:33:53.059 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.057+0000 7fc0b8aa8640 10 monclient: build_initial_monmap 2024-03-25T18:33:53.059 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.057+0000 7fc0b8aa8640 10 monclient: monmap: 2024-03-25T18:33:53.059 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:33:53.059 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:33:53.059 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:33:53.058869+0000 2024-03-25T18:33:53.059 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:33:53.058869+0000 2024-03-25T18:33:53.059 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:53.059 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:33:53.059 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:33:53.059 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:33:53.060 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:33:53.060 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:33:53.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.058+0000 7fc0b8aa8640 1 Processor -- start 2024-03-25T18:33:53.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.058+0000 7fc0b8aa8640 1 -- start start 2024-03-25T18:33:53.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.058+0000 7fc0b8aa8640 10 monclient: init 2024-03-25T18:33:53.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.059+0000 7fc0b8aa8640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:53.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.059+0000 7fc0b8aa8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:33:53.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.059+0000 7fc0b8aa8640 10 monclient(hunting): picked mon.noname-c con 0x7fc0b40b4d60 addr v1:172.21.15.167:6789/0 2024-03-25T18:33:53.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.059+0000 7fc0b8aa8640 10 monclient(hunting): picked mon.noname-a con 0x7fc0b40b1730 addr v1:172.21.15.155:6789/0 2024-03-25T18:33:53.062 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.059+0000 7fc0b8aa8640 10 monclient(hunting): picked mon.noname-b con 0x7fc0b414be90 addr v1:172.21.15.155:6790/0 2024-03-25T18:33:53.062 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.059+0000 7fc0b8aa8640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0b40af940 con 0x7fc0b40b1730 2024-03-25T18:33:53.062 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.059+0000 7fc0b8aa8640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0b40b0a70 con 0x7fc0b40b4d60 2024-03-25T18:33:53.062 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.059+0000 7fc0b8aa8640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0b40b0d50 con 0x7fc0b414be90 2024-03-25T18:33:53.062 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.059+0000 7fc0b8aa8640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:53.062 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.059+0000 7fc0b8aa8640 10 monclient(hunting): authenticate will time out at 8079.369582s 2024-03-25T18:33:53.068 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.067+0000 7fc0b259c640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7fc0b40b4d60 0x7fc0b413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:38230/0 (socket says 172.21.15.155:38230) 2024-03-25T18:33:53.068 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.067+0000 7fc0b259c640 1 -- 172.21.15.155:0/3281145142 learned_addr learned my addr 172.21.15.155:0/3281145142 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:33:53.078 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.077+0000 7fc0b1d9b640 1 -- 172.21.15.155:0/3281145142 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3322009902 0 0) 0x7fc0b40b0a70 con 0x7fc0b40b4d60 2024-03-25T18:33:53.078 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.077+0000 7fc0b1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:33:53.078 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.077+0000 7fc0b1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:53.078 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.077+0000 7fc0b1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:53.078 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.077+0000 7fc0b1d9b640 10 monclient(hunting): my global_id is 4633 2024-03-25T18:33:53.078 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.077+0000 7fc0b1d9b640 1 -- 172.21.15.155:0/3281145142 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0a0003520 con 0x7fc0b40b4d60 2024-03-25T18:33:53.083 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.082+0000 7fc0b1d9b640 1 -- 172.21.15.155:0/3281145142 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 748654179 0 0) 0x7fc0b40b0d50 con 0x7fc0b414be90 2024-03-25T18:33:53.083 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.082+0000 7fc0b1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:33:53.083 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.082+0000 7fc0b1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:53.083 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.082+0000 7fc0b1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:53.083 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.082+0000 7fc0b1d9b640 10 monclient(hunting): my global_id is 4646 2024-03-25T18:33:53.083 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.082+0000 7fc0b1d9b640 1 -- 172.21.15.155:0/3281145142 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0b40b0a70 con 0x7fc0b414be90 2024-03-25T18:33:53.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.089+0000 7fc0b1d9b640 1 -- 172.21.15.155:0/3281145142 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4017726685 0 0) 0x7fc0b40af940 con 0x7fc0b40b1730 2024-03-25T18:33:53.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.089+0000 7fc0b1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:33:53.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.089+0000 7fc0b1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:53.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.089+0000 7fc0b1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:53.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.089+0000 7fc0b1d9b640 10 monclient(hunting): my global_id is 4638 2024-03-25T18:33:53.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.089+0000 7fc0b1d9b640 1 -- 172.21.15.155:0/3281145142 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0b40b0d50 con 0x7fc0b40b1730 2024-03-25T18:33:53.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.101+0000 7fc0b1d9b640 1 -- 172.21.15.155:0/3281145142 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2338294852 0 0) 0x7fc0b40b0d50 con 0x7fc0b40b1730 2024-03-25T18:33:53.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.101+0000 7fc0b1d9b640 1 -- 172.21.15.155:0/3281145142 >> v1:172.21.15.155:6790/0 conn(0x7fc0b414be90 legacy=0x7fc0b414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:53.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.101+0000 7fc0b1d9b640 1 -- 172.21.15.155:0/3281145142 >> v1:172.21.15.167:6789/0 conn(0x7fc0b40b4d60 legacy=0x7fc0b413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:53.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.101+0000 7fc0b1d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:33:53.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.101+0000 7fc0b1d9b640 1 monclient: found mon.noname-a 2024-03-25T18:33:53.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.101+0000 7fc0b1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:53.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.101+0000 7fc0b1d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-25T18:33:53.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.101+0000 7fc0b1d9b640 1 -- 172.21.15.155:0/3281145142 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0b40b1030 con 0x7fc0b40b1730 2024-03-25T18:33:53.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.101+0000 7fc0b1d9b640 10 monclient: _finish_auth 0 2024-03-25T18:33:53.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.101+0000 7fc0b1d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:33:53.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.101+0000 7fc0b1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:53.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.101+0000 7fc0b1d9b640 1 -- 172.21.15.155:0/3281145142 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fc09c0015b0 con 0x7fc0b40b1730 2024-03-25T18:33:53.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.101+0000 7fc0b8aa8640 5 monclient: authenticate success, global_id 4638 2024-03-25T18:33:53.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.101+0000 7fc0b8aa8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:53.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.101+0000 7fc0b1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:53.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.101+0000 7fc0b1d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:33:53.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.101+0000 7fc0b1d9b640 10 monclient: dump: 2024-03-25T18:33:53.104 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:33:53.104 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:33:53.104 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:53.104 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:53.105 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:53.105 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:33:53.105 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:33:53.105 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:33:53.105 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:33:53.105 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:33:53.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.101+0000 7fc0b8aa8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:53.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.130+0000 7fc0b1d9b640 1 -- 172.21.15.155:0/3281145142 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc09c001710 con 0x7fc0b40b1730 2024-03-25T18:33:53.132 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.130+0000 7fc0b1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:53.132 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.130+0000 7fc0b1d9b640 1 -- 172.21.15.155:0/3281145142 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fc09c001b60 con 0x7fc0b40b1730 2024-03-25T18:33:53.132 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.130+0000 7fc0b1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:53.132 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.130+0000 7fc0b1d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:33:53.132 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.130+0000 7fc0b1d9b640 10 monclient: dump: 2024-03-25T18:33:53.132 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:33:53.132 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:33:53.132 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:53.132 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:53.132 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:53.132 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:33:53.132 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:33:53.133 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:33:53.133 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:33:53.133 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:33:53.133 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.130+0000 7fc0b8aa8640 10 monclient: get_monmap_and_config success 2024-03-25T18:33:53.133 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.130+0000 7fc0b8aa8640 4 set_mon_vals no callback set 2024-03-25T18:33:53.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.131+0000 7fc0b8aa8640 10 monclient: shutdown 2024-03-25T18:33:53.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.131+0000 7fc0b8aa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:53.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.131+0000 7fc0b8aa8640 1 -- 172.21.15.155:0/3281145142 >> v1:172.21.15.155:6789/0 conn(0x7fc0b40b1730 legacy=0x7fc0b40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:53.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.131+0000 7fc0b8aa8640 1 -- 172.21.15.155:0/3281145142 shutdown_connections 2024-03-25T18:33:53.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.131+0000 7fc0b8aa8640 1 -- 172.21.15.155:0/3281145142 >> 172.21.15.155:0/3281145142 conn(0x7fc0b40617a0 msgr2=0x7fc0b4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:53.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.131+0000 7fc0b8aa8640 1 -- 172.21.15.155:0/3281145142 shutdown_connections 2024-03-25T18:33:53.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.131+0000 7fc0b8aa8640 1 -- 172.21.15.155:0/3281145142 wait complete. 2024-03-25T18:33:53.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.132+0000 7fc0b8aa8640 10 monclient: build_initial_monmap 2024-03-25T18:33:53.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.132+0000 7fc0b8aa8640 10 monclient: monmap: 2024-03-25T18:33:53.136 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:33:53.136 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:33:53.136 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:33:53.136 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:33:53.136 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:53.137 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:33:53.137 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:33:53.137 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:33:53.137 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:33:53.137 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:33:53.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.132+0000 7fc0b8aa8640 1 librados: starting msgr at 2024-03-25T18:33:53.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.132+0000 7fc0b8aa8640 1 librados: starting objecter 2024-03-25T18:33:53.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.133+0000 7fc0b8aa8640 1 Processor -- start 2024-03-25T18:33:53.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.133+0000 7fc0b8aa8640 1 -- start start 2024-03-25T18:33:53.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.133+0000 7fc0b8aa8640 1 librados: setting wanted keys 2024-03-25T18:33:53.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.133+0000 7fc0b8aa8640 1 librados: calling monclient init 2024-03-25T18:33:53.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.133+0000 7fc0b8aa8640 10 monclient: init 2024-03-25T18:33:53.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.133+0000 7fc0b8aa8640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:53.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.133+0000 7fc0b8aa8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:33:53.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.133+0000 7fc0b8aa8640 10 monclient(hunting): picked mon.noname-c con 0x7fc0b40b1730 addr v1:172.21.15.155:6790/0 2024-03-25T18:33:53.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.133+0000 7fc0b8aa8640 10 monclient(hunting): picked mon.noname-a con 0x7fc0b40b4d60 addr v1:172.21.15.155:6789/0 2024-03-25T18:33:53.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.133+0000 7fc0b8aa8640 10 monclient(hunting): picked mon.noname-b con 0x7fc0b414be90 addr v1:172.21.15.167:6789/0 2024-03-25T18:33:53.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.134+0000 7fc0b8aa8640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0b41324b0 con 0x7fc0b40b4d60 2024-03-25T18:33:53.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.134+0000 7fc0b8aa8640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0b4132640 con 0x7fc0b414be90 2024-03-25T18:33:53.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.134+0000 7fc0b8aa8640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0b41327d0 con 0x7fc0b40b1730 2024-03-25T18:33:53.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.134+0000 7fc0b8aa8640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:53.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.134+0000 7fc0b8aa8640 10 monclient(hunting): authenticate will time out at 8079.443792s 2024-03-25T18:33:53.144 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.142+0000 7fc0b2d9d640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7fc0b414be90 0x7fc0b41d7ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:38292/0 (socket says 172.21.15.155:38292) 2024-03-25T18:33:53.144 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.143+0000 7fc0b2d9d640 1 -- 172.21.15.155:0/3075035102 learned_addr learned my addr 172.21.15.155:0/3075035102 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:33:53.144 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.143+0000 7fc0aaffd640 10 client.?.objecter ms_handle_connect 0x7fc0b414be90 2024-03-25T18:33:53.144 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.143+0000 7fc0aaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:53.149 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.148+0000 7fc0aaffd640 10 client.?.objecter ms_handle_connect 0x7fc0b40b4d60 2024-03-25T18:33:53.149 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.148+0000 7fc0aaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:53.150 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.149+0000 7fc0aaffd640 10 client.?.objecter ms_handle_connect 0x7fc0b40b1730 2024-03-25T18:33:53.150 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.149+0000 7fc0aaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:53.156 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.154+0000 7fc0aaffd640 1 -- 172.21.15.155:0/3075035102 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 816755929 0 0) 0x7fc0b41324b0 con 0x7fc0b40b4d60 2024-03-25T18:33:53.156 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.154+0000 7fc0aaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:53.156 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.154+0000 7fc0aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:53.156 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.154+0000 7fc0aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:53.156 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.154+0000 7fc0aaffd640 10 monclient(hunting): my global_id is 4641 2024-03-25T18:33:53.156 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.155+0000 7fc0aaffd640 1 -- 172.21.15.155:0/3075035102 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc094004eb0 con 0x7fc0b40b4d60 2024-03-25T18:33:53.161 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.160+0000 7fc0aaffd640 1 -- 172.21.15.155:0/3075035102 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2998992832 0 0) 0x7fc0b41327d0 con 0x7fc0b40b1730 2024-03-25T18:33:53.161 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.160+0000 7fc0aaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:53.161 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.160+0000 7fc0aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:53.161 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.160+0000 7fc0aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:53.161 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.160+0000 7fc0aaffd640 10 monclient(hunting): my global_id is 4649 2024-03-25T18:33:53.161 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.160+0000 7fc0aaffd640 1 -- 172.21.15.155:0/3075035102 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0b41324b0 con 0x7fc0b40b1730 2024-03-25T18:33:53.162 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.161+0000 7fc0aaffd640 1 -- 172.21.15.155:0/3075035102 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3836277350 0 0) 0x7fc0b4132640 con 0x7fc0b414be90 2024-03-25T18:33:53.162 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.161+0000 7fc0aaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:53.162 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.161+0000 7fc0aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:53.162 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.161+0000 7fc0aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:53.162 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.161+0000 7fc0aaffd640 10 monclient(hunting): my global_id is 4636 2024-03-25T18:33:53.162 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.161+0000 7fc0aaffd640 1 -- 172.21.15.155:0/3075035102 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0b41327d0 con 0x7fc0b414be90 2024-03-25T18:33:53.181 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.180+0000 7fc0aaffd640 1 -- 172.21.15.155:0/3075035102 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1161596313 0 0) 0x7fc094004eb0 con 0x7fc0b40b4d60 2024-03-25T18:33:53.181 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.180+0000 7fc0aaffd640 1 -- 172.21.15.155:0/3075035102 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc0b4132640 con 0x7fc0b40b4d60 2024-03-25T18:33:53.182 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.180+0000 7fc0aaffd640 1 -- 172.21.15.155:0/3075035102 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fc09c001a20 con 0x7fc0b40b4d60 2024-03-25T18:33:53.182 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.180+0000 7fc0aaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:53.182 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.180+0000 7fc0aaffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:33:53.182 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.180+0000 7fc0aaffd640 10 monclient(hunting): dump: 2024-03-25T18:33:53.182 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:33:53.182 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:33:53.182 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:53.182 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:53.182 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:53.182 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:33:53.182 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:33:53.182 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:33:53.183 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:33:53.183 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:33:53.183 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.180+0000 7fc0aaffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:33:53.185 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.184+0000 7fc0aaffd640 1 -- 172.21.15.155:0/3075035102 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1495740919 0 0) 0x7fc0b41324b0 con 0x7fc0b40b1730 2024-03-25T18:33:53.185 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.184+0000 7fc0aaffd640 1 -- 172.21.15.155:0/3075035102 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc094004eb0 con 0x7fc0b40b1730 2024-03-25T18:33:53.186 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.184+0000 7fc0aaffd640 1 -- 172.21.15.155:0/3075035102 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fc0a4001950 con 0x7fc0b40b1730 2024-03-25T18:33:53.186 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.184+0000 7fc0aaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:53.186 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.184+0000 7fc0aaffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:33:53.186 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.184+0000 7fc0aaffd640 10 monclient(hunting): dump: 2024-03-25T18:33:53.186 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:33:53.186 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:33:53.186 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:53.186 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:53.186 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:53.186 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:33:53.186 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:33:53.186 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:33:53.187 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:33:53.187 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:33:53.187 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.186+0000 7fc0aaffd640 1 -- 172.21.15.155:0/3075035102 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1073510579 0 0) 0x7fc0b41327d0 con 0x7fc0b414be90 2024-03-25T18:33:53.187 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.186+0000 7fc0aaffd640 1 -- 172.21.15.155:0/3075035102 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc0b41324b0 con 0x7fc0b414be90 2024-03-25T18:33:53.187 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.186+0000 7fc0aaffd640 1 -- 172.21.15.155:0/3075035102 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fc0ac001bd0 con 0x7fc0b414be90 2024-03-25T18:33:53.188 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.186+0000 7fc0aaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:53.188 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.186+0000 7fc0aaffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:33:53.188 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.186+0000 7fc0aaffd640 10 monclient(hunting): dump: 2024-03-25T18:33:53.188 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:33:53.188 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:33:53.188 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:53.188 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:53.188 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:53.188 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:33:53.188 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:33:53.188 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:33:53.188 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:33:53.188 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:33:53.199 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.198+0000 7fc0aaffd640 1 -- 172.21.15.155:0/3075035102 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2757924246 0 0) 0x7fc094004eb0 con 0x7fc0b40b1730 2024-03-25T18:33:53.200 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.198+0000 7fc0aaffd640 1 -- 172.21.15.155:0/3075035102 >> v1:172.21.15.167:6789/0 conn(0x7fc0b414be90 legacy=0x7fc0b41d7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:53.200 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.198+0000 7fc0aaffd640 1 -- 172.21.15.155:0/3075035102 >> v1:172.21.15.155:6789/0 conn(0x7fc0b40b4d60 legacy=0x7fc0b41d78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:53.200 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.198+0000 7fc0aaffd640 10 monclient: _finish_hunting 0 2024-03-25T18:33:53.200 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.198+0000 7fc0aaffd640 1 monclient: found mon.c 2024-03-25T18:33:53.200 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.198+0000 7fc0aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:53.200 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.198+0000 7fc0aaffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:33:53.200 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.198+0000 7fc0aaffd640 1 -- 172.21.15.155:0/3075035102 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0b4138e60 con 0x7fc0b40b1730 2024-03-25T18:33:53.200 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.198+0000 7fc0aaffd640 10 monclient: _finish_auth 0 2024-03-25T18:33:53.200 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.198+0000 7fc0aaffd640 10 monclient: _check_auth_tickets 2024-03-25T18:33:53.200 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.198+0000 7fc0aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:53.200 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.198+0000 7fc0b8aa8640 5 monclient: authenticate success, global_id 4649 2024-03-25T18:33:53.200 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.198+0000 7fc0b8aa8640 10 monclient: _renew_subs 2024-03-25T18:33:53.201 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.198+0000 7fc0b8aa8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:33:53.201 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.198+0000 7fc0b8aa8640 1 -- 172.21.15.155:0/3075035102 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0b4139070 con 0x7fc0b40b1730 2024-03-25T18:33:53.201 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.198+0000 7fc0b8aa8640 10 client.4649.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:33:53.201 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.198+0000 7fc0b8aa8640 10 monclient: _renew_subs 2024-03-25T18:33:53.201 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.198+0000 7fc0b8aa8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:33:53.201 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.198+0000 7fc0b8aa8640 1 -- 172.21.15.155:0/3075035102 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0b4137a80 con 0x7fc0b40b1730 2024-03-25T18:33:53.201 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.198+0000 7fc0b8aa8640 1 librados: init done 2024-03-25T18:33:53.202 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.199+0000 7fc0b8aa8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:53.202 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.199+0000 7fc0b8aa8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:53.202 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.199+0000 7fc0b8aa8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:33:53.202 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.199+0000 7fc0b8aa8640 1 -- 172.21.15.155:0/3075035102 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0b41d7730 con 0x7fc0b40b1730 2024-03-25T18:33:53.503 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.501+0000 7fc0aaffd640 1 -- 172.21.15.155:0/3075035102 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc0a4001d60 con 0x7fc0b40b1730 2024-03-25T18:33:53.503 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.502+0000 7fc0aaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:53.503 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.502+0000 7fc0b159a640 4 set_mon_vals no callback set 2024-03-25T18:33:53.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.502+0000 7fc0aaffd640 1 -- 172.21.15.155:0/3075035102 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fc0a4004980 con 0x7fc0b40b1730 2024-03-25T18:33:53.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.502+0000 7fc0aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:53.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.502+0000 7fc0aaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:33:53.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.502+0000 7fc0aaffd640 10 monclient: dump: 2024-03-25T18:33:53.504 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:33:53.504 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:33:53.504 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:53.504 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:33:53.504 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:53.504 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:33:53.505 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:33:53.505 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:33:53.505 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:33:53.505 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:33:53.505 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.503+0000 7fc0aaffd640 1 -- 172.21.15.155:0/3075035102 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7fc0a401a990 con 0x7fc0b40b1730 2024-03-25T18:33:53.505 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.503+0000 7fc0abfff640 10 monclient: get_auth_request con 0x7fc09406e0c0 auth_method 0 2024-03-25T18:33:53.505 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.503+0000 7fc0aaffd640 10 client.4649.objecter ms_handle_connect 0x7fc09406e0c0 2024-03-25T18:33:53.512 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.511+0000 7fc0aaffd640 1 -- 172.21.15.155:0/3075035102 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7fc0a40547d0 con 0x7fc0b40b1730 2024-03-25T18:33:53.512 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.511+0000 7fc0aaffd640 10 client.4649.objecter ms_dispatch 0x7fc0b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:33:53.512 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.511+0000 7fc0aaffd640 3 client.4649.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:33:53.512 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.511+0000 7fc0aaffd640 3 client.4649.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:33:53.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.511+0000 7fc0aaffd640 20 client.4649.objecter dump_active .. 0 homeless 2024-03-25T18:33:53.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.511+0000 7fc0aaffd640 1 -- 172.21.15.155:0/3075035102 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc0a4050110 con 0x7fc0b40b1730 2024-03-25T18:33:53.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.511+0000 7fc0aaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:53.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.511+0000 7fc0aaffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:33:53.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.756+0000 7fc0b8aa8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:53.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.756+0000 7fc0b8aa8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:53.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.756+0000 7fc0b8aa8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:33:53.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.756+0000 7fc0b8aa8640 1 -- 172.21.15.155:0/3075035102 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc0b413a950 con 0x7fc0b40b1730 2024-03-25T18:33:53.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.782+0000 7fc0aaffd640 1 -- 172.21.15.155:0/3075035102 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7fc0b413a950 con 0x7fc0b40b1730 2024-03-25T18:33:53.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.782+0000 7fc0aaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:53.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.782+0000 7fc0aaffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:33:53.784 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:33:53.784 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:33:53.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.784+0000 7fc0b8aa8640 10 librados: watch_flush enter 2024-03-25T18:33:53.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.784+0000 7fc0b8aa8640 10 librados: watch_flush exit 2024-03-25T18:33:53.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.784+0000 7fc0b8aa8640 20 client.4649.objecter shutdown clearing up homeless session... 2024-03-25T18:33:53.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.784+0000 7fc0b8aa8640 10 client.4649.objecter successfully canceled tick 2024-03-25T18:33:53.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.785+0000 7fc0b8aa8640 1 -- 172.21.15.155:0/3075035102 >> v1:172.21.15.167:6800/2919070503 conn(0x7fc09406e0c0 legacy=0x7fc094070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:53.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.785+0000 7fc0b8aa8640 10 monclient: shutdown 2024-03-25T18:33:53.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.785+0000 7fc0b8aa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:53.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.785+0000 7fc0b8aa8640 1 -- 172.21.15.155:0/3075035102 >> v1:172.21.15.155:6790/0 conn(0x7fc0b40b1730 legacy=0x7fc0b4144890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:53.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.785+0000 7fc0b8aa8640 1 -- 172.21.15.155:0/3075035102 shutdown_connections 2024-03-25T18:33:53.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.785+0000 7fc0b8aa8640 1 -- 172.21.15.155:0/3075035102 >> 172.21.15.155:0/3075035102 conn(0x7fc0b40617a0 msgr2=0x7fc0b40b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:53.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.785+0000 7fc0b8aa8640 1 -- 172.21.15.155:0/3075035102 shutdown_connections 2024-03-25T18:33:53.788 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.785+0000 7fc0b8aa8640 1 -- 172.21.15.155:0/3075035102 wait complete. 2024-03-25T18:33:53.788 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:33:53.785+0000 7fc0b8aa8640 1 librados: shutdown 2024-03-25T18:33:53.799 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:34:00.605 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:34:00.718 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.717+0000 7f979eba8640 10 monclient: get_monmap_and_config 2024-03-25T18:34:00.718 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.717+0000 7f979eba8640 10 monclient: build_initial_monmap 2024-03-25T18:34:00.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.717+0000 7f979eba8640 10 monclient: monmap: 2024-03-25T18:34:00.719 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:34:00.719 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:34:00.719 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:34:00.718491+0000 2024-03-25T18:34:00.719 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:34:00.718491+0000 2024-03-25T18:34:00.719 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:00.719 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:34:00.719 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:34:00.719 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:34:00.719 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:34:00.719 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:34:00.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.718+0000 7f979eba8640 1 Processor -- start 2024-03-25T18:34:00.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.718+0000 7f979eba8640 1 -- start start 2024-03-25T18:34:00.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.718+0000 7f979eba8640 10 monclient: init 2024-03-25T18:34:00.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.719+0000 7f979eba8640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:00.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.719+0000 7f979eba8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:34:00.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.719+0000 7f979eba8640 10 monclient(hunting): picked mon.noname-a con 0x7f97980a98f0 addr v1:172.21.15.155:6789/0 2024-03-25T18:34:00.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.719+0000 7f979eba8640 10 monclient(hunting): picked mon.noname-c con 0x7f979813f710 addr v1:172.21.15.167:6789/0 2024-03-25T18:34:00.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.719+0000 7f979eba8640 10 monclient(hunting): picked mon.noname-b con 0x7f97981430c0 addr v1:172.21.15.155:6790/0 2024-03-25T18:34:00.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.719+0000 7f979eba8640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9798142520 con 0x7f97980a98f0 2024-03-25T18:34:00.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.719+0000 7f979eba8640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97980b28a0 con 0x7f979813f710 2024-03-25T18:34:00.721 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.719+0000 7f979eba8640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97980b2a30 con 0x7f97981430c0 2024-03-25T18:34:00.721 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.719+0000 7f979eba8640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:00.721 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.719+0000 7f979eba8640 10 monclient(hunting): authenticate will time out at 8087.028944s 2024-03-25T18:34:00.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.723+0000 7f979d145640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f97981430c0 0x7f979814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:36674/0 (socket says 172.21.15.155:36674) 2024-03-25T18:34:00.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.723+0000 7f979d145640 1 -- 172.21.15.155:0/845994562 learned_addr learned my addr 172.21.15.155:0/845994562 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:34:00.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.736+0000 7f97977fe640 1 -- 172.21.15.155:0/845994562 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1628363322 0 0) 0x7f97980b2a30 con 0x7f97981430c0 2024-03-25T18:34:00.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.736+0000 7f97977fe640 10 monclient(hunting): _negotiate 2024-03-25T18:34:00.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.736+0000 7f97977fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:00.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.736+0000 7f97977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:00.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.736+0000 7f97977fe640 10 monclient(hunting): my global_id is 4652 2024-03-25T18:34:00.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.736+0000 7f97977fe640 1 -- 172.21.15.155:0/845994562 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9788003520 con 0x7f97981430c0 2024-03-25T18:34:00.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.737+0000 7f97977fe640 1 -- 172.21.15.155:0/845994562 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 92948382 0 0) 0x7f97980b28a0 con 0x7f979813f710 2024-03-25T18:34:00.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.737+0000 7f97977fe640 10 monclient(hunting): _negotiate 2024-03-25T18:34:00.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.737+0000 7f97977fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:00.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.737+0000 7f97977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:00.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.737+0000 7f97977fe640 10 monclient(hunting): my global_id is 4639 2024-03-25T18:34:00.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.737+0000 7f97977fe640 1 -- 172.21.15.155:0/845994562 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97980b2a30 con 0x7f979813f710 2024-03-25T18:34:00.756 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.755+0000 7f97977fe640 1 -- 172.21.15.155:0/845994562 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2821554063 0 0) 0x7f9788003520 con 0x7f97981430c0 2024-03-25T18:34:00.756 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.755+0000 7f97977fe640 1 -- 172.21.15.155:0/845994562 >> v1:172.21.15.167:6789/0 conn(0x7f979813f710 legacy=0x7f97981429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:00.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.755+0000 7f97977fe640 1 -- 172.21.15.155:0/845994562 >> v1:172.21.15.155:6789/0 conn(0x7f97980a98f0 legacy=0x7f97980a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:34:00.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.755+0000 7f97977fe640 10 monclient: _finish_hunting 0 2024-03-25T18:34:00.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.755+0000 7f97977fe640 1 monclient: found mon.noname-b 2024-03-25T18:34:00.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.755+0000 7f97977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:00.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.755+0000 7f97977fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-25T18:34:00.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.755+0000 7f97977fe640 1 -- 172.21.15.155:0/845994562 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97980b44c0 con 0x7f97981430c0 2024-03-25T18:34:00.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.755+0000 7f97977fe640 10 monclient: _finish_auth 0 2024-03-25T18:34:00.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.755+0000 7f97977fe640 10 monclient: _check_auth_tickets 2024-03-25T18:34:00.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.755+0000 7f97977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:00.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.755+0000 7f97977fe640 1 -- 172.21.15.155:0/845994562 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f9790001a10 con 0x7f97981430c0 2024-03-25T18:34:00.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.755+0000 7f97977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:00.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.755+0000 7f97977fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:34:00.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.755+0000 7f97977fe640 10 monclient: dump: 2024-03-25T18:34:00.757 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:34:00.757 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:34:00.758 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:00.758 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:00.758 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:00.758 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:34:00.758 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:34:00.758 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:34:00.758 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:34:00.758 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:34:00.758 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.755+0000 7f979eba8640 5 monclient: authenticate success, global_id 4652 2024-03-25T18:34:00.758 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.755+0000 7f979eba8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:00.761 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.760+0000 7f97977fe640 1 -- 172.21.15.155:0/845994562 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9790001b70 con 0x7f97981430c0 2024-03-25T18:34:00.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.760+0000 7f97977fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:00.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.760+0000 7f97977fe640 1 -- 172.21.15.155:0/845994562 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f9790001f80 con 0x7f97981430c0 2024-03-25T18:34:00.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.760+0000 7f97977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:00.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.760+0000 7f97977fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:34:00.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.760+0000 7f97977fe640 10 monclient: dump: 2024-03-25T18:34:00.762 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:34:00.762 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:34:00.762 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:00.762 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:00.762 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:00.762 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:34:00.762 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:34:00.762 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:34:00.763 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:34:00.763 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:34:00.763 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.760+0000 7f979eba8640 10 monclient: get_monmap_and_config success 2024-03-25T18:34:00.763 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.760+0000 7f979eba8640 4 set_mon_vals no callback set 2024-03-25T18:34:00.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.761+0000 7f979eba8640 10 monclient: shutdown 2024-03-25T18:34:00.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.761+0000 7f979eba8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:00.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.761+0000 7f979eba8640 1 -- 172.21.15.155:0/845994562 >> v1:172.21.15.155:6790/0 conn(0x7f97981430c0 legacy=0x7f979814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:00.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.761+0000 7f979eba8640 1 -- 172.21.15.155:0/845994562 shutdown_connections 2024-03-25T18:34:00.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.761+0000 7f979eba8640 1 -- 172.21.15.155:0/845994562 >> 172.21.15.155:0/845994562 conn(0x7f97980b7ea0 msgr2=0x7f97980b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:00.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.761+0000 7f979eba8640 1 -- 172.21.15.155:0/845994562 shutdown_connections 2024-03-25T18:34:00.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.761+0000 7f979eba8640 1 -- 172.21.15.155:0/845994562 wait complete. 2024-03-25T18:34:00.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.762+0000 7f979eba8640 10 monclient: build_initial_monmap 2024-03-25T18:34:00.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.762+0000 7f979eba8640 10 monclient: monmap: 2024-03-25T18:34:00.765 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:34:00.765 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:34:00.765 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:34:00.765 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:34:00.765 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:00.765 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:34:00.765 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:34:00.765 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:34:00.765 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:34:00.765 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:34:00.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.763+0000 7f979eba8640 1 librados: starting msgr at 2024-03-25T18:34:00.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.763+0000 7f979eba8640 1 librados: starting objecter 2024-03-25T18:34:00.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.763+0000 7f979eba8640 1 Processor -- start 2024-03-25T18:34:00.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.763+0000 7f979eba8640 1 -- start start 2024-03-25T18:34:00.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.763+0000 7f979eba8640 1 librados: setting wanted keys 2024-03-25T18:34:00.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.763+0000 7f979eba8640 1 librados: calling monclient init 2024-03-25T18:34:00.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.763+0000 7f979eba8640 10 monclient: init 2024-03-25T18:34:00.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.764+0000 7f979eba8640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:00.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.764+0000 7f979eba8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:34:00.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.764+0000 7f979eba8640 10 monclient(hunting): picked mon.noname-a con 0x7f97980a98f0 addr v1:172.21.15.155:6789/0 2024-03-25T18:34:00.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.764+0000 7f979eba8640 10 monclient(hunting): picked mon.noname-c con 0x7f979813f710 addr v1:172.21.15.155:6790/0 2024-03-25T18:34:00.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.764+0000 7f979eba8640 10 monclient(hunting): picked mon.noname-b con 0x7f97981430c0 addr v1:172.21.15.167:6789/0 2024-03-25T18:34:00.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.764+0000 7f979eba8640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97981373f0 con 0x7f97980a98f0 2024-03-25T18:34:00.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.764+0000 7f979eba8640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97981db280 con 0x7f97981430c0 2024-03-25T18:34:00.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.764+0000 7f979eba8640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9798138f40 con 0x7f979813f710 2024-03-25T18:34:00.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.764+0000 7f979eba8640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:00.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.764+0000 7f979eba8640 10 monclient(hunting): authenticate will time out at 8087.074067s 2024-03-25T18:34:00.772 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.771+0000 7f979c944640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f97980a98f0 0x7f97981dab70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:40346/0 (socket says 172.21.15.155:40346) 2024-03-25T18:34:00.772 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.771+0000 7f979c944640 1 -- 172.21.15.155:0/2183521656 learned_addr learned my addr 172.21.15.155:0/2183521656 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:34:00.773 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.772+0000 7f97957fa640 10 client.?.objecter ms_handle_connect 0x7f97980a98f0 2024-03-25T18:34:00.773 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.772+0000 7f97957fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:00.774 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.773+0000 7f97957fa640 10 client.?.objecter ms_handle_connect 0x7f97981430c0 2024-03-25T18:34:00.774 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.773+0000 7f97957fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:00.779 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.779+0000 7f97957fa640 10 client.?.objecter ms_handle_connect 0x7f979813f710 2024-03-25T18:34:00.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.779+0000 7f97957fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:00.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.785+0000 7f97957fa640 1 -- 172.21.15.155:0/2183521656 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1481414620 0 0) 0x7f97981373f0 con 0x7f97980a98f0 2024-03-25T18:34:00.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.785+0000 7f97957fa640 10 monclient(hunting): _negotiate 2024-03-25T18:34:00.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.785+0000 7f97957fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:00.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.785+0000 7f97957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:00.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.785+0000 7f97957fa640 10 monclient(hunting): my global_id is 4644 2024-03-25T18:34:00.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.786+0000 7f97957fa640 1 -- 172.21.15.155:0/2183521656 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f977c004eb0 con 0x7f97980a98f0 2024-03-25T18:34:00.788 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.787+0000 7f97957fa640 1 -- 172.21.15.155:0/2183521656 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1617811305 0 0) 0x7f97981db280 con 0x7f97981430c0 2024-03-25T18:34:00.788 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.787+0000 7f97957fa640 10 monclient(hunting): _negotiate 2024-03-25T18:34:00.788 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.787+0000 7f97957fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:00.788 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.787+0000 7f97957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:00.789 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.787+0000 7f97957fa640 10 monclient(hunting): my global_id is 4642 2024-03-25T18:34:00.789 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.787+0000 7f97957fa640 1 -- 172.21.15.155:0/2183521656 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97981373f0 con 0x7f97981430c0 2024-03-25T18:34:00.790 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.789+0000 7f97957fa640 1 -- 172.21.15.155:0/2183521656 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3234647573 0 0) 0x7f9798138f40 con 0x7f979813f710 2024-03-25T18:34:00.791 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.789+0000 7f97957fa640 10 monclient(hunting): _negotiate 2024-03-25T18:34:00.791 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.789+0000 7f97957fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:00.791 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.789+0000 7f97957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:00.791 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.789+0000 7f97957fa640 10 monclient(hunting): my global_id is 4655 2024-03-25T18:34:00.791 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.789+0000 7f97957fa640 1 -- 172.21.15.155:0/2183521656 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97981db280 con 0x7f979813f710 2024-03-25T18:34:00.800 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.799+0000 7f97957fa640 1 -- 172.21.15.155:0/2183521656 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 766047288 0 0) 0x7f97981db280 con 0x7f979813f710 2024-03-25T18:34:00.800 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.799+0000 7f97957fa640 1 -- 172.21.15.155:0/2183521656 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9798138f40 con 0x7f979813f710 2024-03-25T18:34:00.800 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.799+0000 7f97957fa640 1 -- 172.21.15.155:0/2183521656 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f9784001930 con 0x7f979813f710 2024-03-25T18:34:00.800 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.799+0000 7f97957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:00.800 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.799+0000 7f97957fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:34:00.800 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.799+0000 7f97957fa640 10 monclient(hunting): dump: 2024-03-25T18:34:00.800 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:34:00.800 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:34:00.800 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:00.800 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:00.800 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:00.801 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:34:00.801 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:34:00.801 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:34:00.801 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:34:00.801 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:34:00.801 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.799+0000 7f97957fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:34:00.807 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.806+0000 7f97957fa640 1 -- 172.21.15.155:0/2183521656 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3535577470 0 0) 0x7f97981373f0 con 0x7f97981430c0 2024-03-25T18:34:00.808 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.806+0000 7f97957fa640 1 -- 172.21.15.155:0/2183521656 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f97981db280 con 0x7f97981430c0 2024-03-25T18:34:00.808 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.806+0000 7f97957fa640 1 -- 172.21.15.155:0/2183521656 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f9790001e80 con 0x7f97981430c0 2024-03-25T18:34:00.808 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.806+0000 7f97957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:00.808 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.806+0000 7f97957fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:00.808 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.806+0000 7f97957fa640 10 monclient(hunting): dump: 2024-03-25T18:34:00.808 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:34:00.808 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:34:00.808 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:00.808 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:00.808 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:00.808 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:34:00.808 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:34:00.808 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:34:00.808 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:34:00.808 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:34:00.811 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.809+0000 7f97957fa640 1 -- 172.21.15.155:0/2183521656 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 984541590 0 0) 0x7f9798138f40 con 0x7f979813f710 2024-03-25T18:34:00.811 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.809+0000 7f97957fa640 1 -- 172.21.15.155:0/2183521656 >> v1:172.21.15.167:6789/0 conn(0x7f97981430c0 legacy=0x7f97981332c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:00.811 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.809+0000 7f97957fa640 1 -- 172.21.15.155:0/2183521656 >> v1:172.21.15.155:6789/0 conn(0x7f97980a98f0 legacy=0x7f97981dab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:00.811 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.809+0000 7f97957fa640 10 monclient: _finish_hunting 0 2024-03-25T18:34:00.811 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.809+0000 7f97957fa640 1 monclient: found mon.c 2024-03-25T18:34:00.811 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.809+0000 7f97957fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:00.811 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.809+0000 7f97957fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:34:00.811 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.809+0000 7f97957fa640 1 -- 172.21.15.155:0/2183521656 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97981339d0 con 0x7f979813f710 2024-03-25T18:34:00.811 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.809+0000 7f97957fa640 10 monclient: _finish_auth 0 2024-03-25T18:34:00.812 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.809+0000 7f97957fa640 10 monclient: _check_auth_tickets 2024-03-25T18:34:00.812 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.809+0000 7f97957fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:00.812 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.809+0000 7f979eba8640 5 monclient: authenticate success, global_id 4655 2024-03-25T18:34:00.812 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.809+0000 7f979eba8640 10 monclient: _renew_subs 2024-03-25T18:34:00.812 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.809+0000 7f979eba8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:34:00.812 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.809+0000 7f979eba8640 1 -- 172.21.15.155:0/2183521656 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9798137f50 con 0x7f979813f710 2024-03-25T18:34:00.812 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.809+0000 7f979eba8640 10 client.4655.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:34:00.812 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.809+0000 7f979eba8640 10 monclient: _renew_subs 2024-03-25T18:34:00.812 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.809+0000 7f979eba8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:34:00.812 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.809+0000 7f979eba8640 1 -- 172.21.15.155:0/2183521656 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f979813c550 con 0x7f979813f710 2024-03-25T18:34:00.812 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.809+0000 7f979eba8640 1 librados: init done 2024-03-25T18:34:00.813 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.811+0000 7f979eba8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:00.813 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.811+0000 7f979eba8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:00.813 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.811+0000 7f979eba8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:34:00.813 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:00.811+0000 7f979eba8640 1 -- 172.21.15.155:0/2183521656 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97980b3ef0 con 0x7f979813f710 2024-03-25T18:34:01.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:01.031+0000 7f97957fa640 1 -- 172.21.15.155:0/2183521656 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f97840035f0 con 0x7f979813f710 2024-03-25T18:34:01.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:01.031+0000 7f97957fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:01.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:01.031+0000 7f9796ffd640 4 set_mon_vals no callback set 2024-03-25T18:34:01.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:01.109+0000 7f97957fa640 1 -- 172.21.15.155:0/2183521656 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f9784001d80 con 0x7f979813f710 2024-03-25T18:34:01.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:01.109+0000 7f97957fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:01.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:01.109+0000 7f97957fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:34:01.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:01.109+0000 7f97957fa640 10 monclient: dump: 2024-03-25T18:34:01.110 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:34:01.110 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:34:01.110 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:01.111 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:01.111 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:01.111 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:34:01.111 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:34:01.111 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:34:01.111 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:34:01.111 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:34:01.111 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:01.109+0000 7f97957fa640 1 -- 172.21.15.155:0/2183521656 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7f9784019830 con 0x7f979813f710 2024-03-25T18:34:01.112 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:01.110+0000 7f979c944640 10 monclient: get_auth_request con 0x7f977c06dee0 auth_method 0 2024-03-25T18:34:01.112 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:01.110+0000 7f97957fa640 10 client.4655.objecter ms_handle_connect 0x7f977c06dee0 2024-03-25T18:34:01.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:01.116+0000 7f97957fa640 1 -- 172.21.15.155:0/2183521656 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7f97840535f0 con 0x7f979813f710 2024-03-25T18:34:01.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:01.116+0000 7f97957fa640 10 client.4655.objecter ms_dispatch 0x7f9798000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:34:01.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:01.116+0000 7f97957fa640 3 client.4655.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:34:01.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:01.116+0000 7f97957fa640 3 client.4655.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:34:01.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:01.116+0000 7f97957fa640 20 client.4655.objecter dump_active .. 0 homeless 2024-03-25T18:34:01.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:01.116+0000 7f97957fa640 1 -- 172.21.15.155:0/2183521656 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f978404ef30 con 0x7f979813f710 2024-03-25T18:34:01.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:01.116+0000 7f97957fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:01.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:01.116+0000 7f97957fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:34:01.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:01.341+0000 7f979eba8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:01.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:01.341+0000 7f979eba8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:01.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:01.341+0000 7f979eba8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:34:01.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:01.341+0000 7f979eba8640 1 -- 172.21.15.155:0/2183521656 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f97981375a0 con 0x7f979813f710 2024-03-25T18:34:01.444 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:01.443+0000 7f97957fa640 1 -- 172.21.15.155:0/2183521656 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7f97981375a0 con 0x7f979813f710 2024-03-25T18:34:01.444 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:01.443+0000 7f97957fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:01.444 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:01.443+0000 7f97957fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:34:01.444 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:34:01.444 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:34:01.447 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:01.445+0000 7f979eba8640 10 librados: watch_flush enter 2024-03-25T18:34:01.447 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:01.445+0000 7f979eba8640 10 librados: watch_flush exit 2024-03-25T18:34:01.447 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:01.445+0000 7f979eba8640 20 client.4655.objecter shutdown clearing up homeless session... 2024-03-25T18:34:01.447 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:01.445+0000 7f979eba8640 10 client.4655.objecter successfully canceled tick 2024-03-25T18:34:01.447 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:01.445+0000 7f979eba8640 1 -- 172.21.15.155:0/2183521656 >> v1:172.21.15.167:6800/2919070503 conn(0x7f977c06dee0 legacy=0x7f977c0703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:01.447 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:01.445+0000 7f979eba8640 10 monclient: shutdown 2024-03-25T18:34:01.447 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:01.445+0000 7f979eba8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:01.447 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:01.445+0000 7f979eba8640 1 -- 172.21.15.155:0/2183521656 >> v1:172.21.15.155:6790/0 conn(0x7f979813f710 legacy=0x7f9798132bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:01.447 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:01.445+0000 7f979eba8640 1 -- 172.21.15.155:0/2183521656 shutdown_connections 2024-03-25T18:34:01.448 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:01.445+0000 7f979eba8640 1 -- 172.21.15.155:0/2183521656 >> 172.21.15.155:0/2183521656 conn(0x7f97980b7ea0 msgr2=0x7f9798074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:01.448 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:01.445+0000 7f979eba8640 1 -- 172.21.15.155:0/2183521656 shutdown_connections 2024-03-25T18:34:01.448 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:01.446+0000 7f979eba8640 1 -- 172.21.15.155:0/2183521656 wait complete. 2024-03-25T18:34:01.448 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:01.446+0000 7f979eba8640 1 librados: shutdown 2024-03-25T18:34:01.458 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:34:08.265 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:34:08.380 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.379+0000 7f1f865d6640 10 monclient: get_monmap_and_config 2024-03-25T18:34:08.380 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.379+0000 7f1f865d6640 10 monclient: build_initial_monmap 2024-03-25T18:34:08.380 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.379+0000 7f1f865d6640 10 monclient: monmap: 2024-03-25T18:34:08.381 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:34:08.381 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:34:08.381 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:34:08.380699+0000 2024-03-25T18:34:08.381 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:34:08.380699+0000 2024-03-25T18:34:08.381 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:08.381 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:34:08.381 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:34:08.381 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:34:08.381 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:34:08.381 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:34:08.382 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.380+0000 7f1f865d6640 1 Processor -- start 2024-03-25T18:34:08.382 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.380+0000 7f1f865d6640 1 -- start start 2024-03-25T18:34:08.382 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.380+0000 7f1f865d6640 10 monclient: init 2024-03-25T18:34:08.382 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.381+0000 7f1f865d6640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:08.382 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.381+0000 7f1f865d6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:34:08.382 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.381+0000 7f1f865d6640 10 monclient(hunting): picked mon.noname-c con 0x7f1f800a98f0 addr v1:172.21.15.167:6789/0 2024-03-25T18:34:08.382 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.381+0000 7f1f865d6640 10 monclient(hunting): picked mon.noname-b con 0x7f1f8013f710 addr v1:172.21.15.155:6790/0 2024-03-25T18:34:08.382 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.381+0000 7f1f865d6640 10 monclient(hunting): picked mon.noname-a con 0x7f1f801430c0 addr v1:172.21.15.155:6789/0 2024-03-25T18:34:08.383 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.381+0000 7f1f865d6640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f80142520 con 0x7f1f801430c0 2024-03-25T18:34:08.383 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.381+0000 7f1f865d6640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f800b28a0 con 0x7f1f800a98f0 2024-03-25T18:34:08.383 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.381+0000 7f1f865d6640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f800b2a30 con 0x7f1f8013f710 2024-03-25T18:34:08.383 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.381+0000 7f1f865d6640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:08.383 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.381+0000 7f1f865d6640 10 monclient(hunting): authenticate will time out at 8094.691028s 2024-03-25T18:34:08.391 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.390+0000 7f1f84b73640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f1f801430c0 0x7f1f8014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:45652/0 (socket says 172.21.15.155:45652) 2024-03-25T18:34:08.392 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.390+0000 7f1f84b73640 1 -- 172.21.15.155:0/2598893275 learned_addr learned my addr 172.21.15.155:0/2598893275 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:34:08.403 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.401+0000 7f1f7effd640 1 -- 172.21.15.155:0/2598893275 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1680076499 0 0) 0x7f1f80142520 con 0x7f1f801430c0 2024-03-25T18:34:08.403 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.401+0000 7f1f7effd640 10 monclient(hunting): _negotiate 2024-03-25T18:34:08.403 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.401+0000 7f1f7effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:08.403 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.401+0000 7f1f7effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:08.403 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.401+0000 7f1f7effd640 10 monclient(hunting): my global_id is 4647 2024-03-25T18:34:08.403 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.401+0000 7f1f7effd640 1 -- 172.21.15.155:0/2598893275 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f68003520 con 0x7f1f801430c0 2024-03-25T18:34:08.411 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.409+0000 7f1f7effd640 1 -- 172.21.15.155:0/2598893275 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 571715894 0 0) 0x7f1f68003520 con 0x7f1f801430c0 2024-03-25T18:34:08.411 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.409+0000 7f1f7effd640 1 -- 172.21.15.155:0/2598893275 >> v1:172.21.15.155:6790/0 conn(0x7f1f8013f710 legacy=0x7f1f801429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:08.411 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.409+0000 7f1f7effd640 1 -- 172.21.15.155:0/2598893275 >> v1:172.21.15.167:6789/0 conn(0x7f1f800a98f0 legacy=0x7f1f800a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:08.411 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.409+0000 7f1f7effd640 10 monclient: _finish_hunting 0 2024-03-25T18:34:08.411 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.409+0000 7f1f7effd640 1 monclient: found mon.noname-a 2024-03-25T18:34:08.411 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.409+0000 7f1f7effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:08.411 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.409+0000 7f1f7effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-25T18:34:08.411 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.409+0000 7f1f7effd640 1 -- 172.21.15.155:0/2598893275 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f800b44c0 con 0x7f1f801430c0 2024-03-25T18:34:08.411 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.409+0000 7f1f7effd640 10 monclient: _finish_auth 0 2024-03-25T18:34:08.411 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.409+0000 7f1f7effd640 10 monclient: _check_auth_tickets 2024-03-25T18:34:08.411 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.409+0000 7f1f7effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:08.411 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.409+0000 7f1f7effd640 1 -- 172.21.15.155:0/2598893275 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f1f78001a10 con 0x7f1f801430c0 2024-03-25T18:34:08.411 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.409+0000 7f1f7effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:08.411 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.409+0000 7f1f7effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:34:08.412 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.409+0000 7f1f7effd640 10 monclient: dump: 2024-03-25T18:34:08.412 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:34:08.412 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:34:08.412 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:08.412 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:08.412 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:08.412 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:34:08.412 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:34:08.412 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:34:08.412 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:34:08.412 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:34:08.412 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.409+0000 7f1f865d6640 5 monclient: authenticate success, global_id 4647 2024-03-25T18:34:08.412 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.409+0000 7f1f865d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:08.416 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.415+0000 7f1f7effd640 1 -- 172.21.15.155:0/2598893275 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1f78001b70 con 0x7f1f801430c0 2024-03-25T18:34:08.416 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.415+0000 7f1f7effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:08.417 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.415+0000 7f1f7effd640 1 -- 172.21.15.155:0/2598893275 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f1f78001f80 con 0x7f1f801430c0 2024-03-25T18:34:08.417 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.415+0000 7f1f7effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:08.417 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.415+0000 7f1f7effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:34:08.417 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.415+0000 7f1f7effd640 10 monclient: dump: 2024-03-25T18:34:08.417 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:34:08.417 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:34:08.417 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:08.417 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:08.417 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:08.417 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:34:08.417 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:34:08.417 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:34:08.417 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:34:08.417 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:34:08.417 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.415+0000 7f1f865d6640 10 monclient: get_monmap_and_config success 2024-03-25T18:34:08.418 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.415+0000 7f1f865d6640 4 set_mon_vals no callback set 2024-03-25T18:34:08.418 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.416+0000 7f1f865d6640 10 monclient: shutdown 2024-03-25T18:34:08.419 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.416+0000 7f1f865d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:08.419 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.416+0000 7f1f865d6640 1 -- 172.21.15.155:0/2598893275 >> v1:172.21.15.155:6789/0 conn(0x7f1f801430c0 legacy=0x7f1f8014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:08.419 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.416+0000 7f1f865d6640 1 -- 172.21.15.155:0/2598893275 shutdown_connections 2024-03-25T18:34:08.419 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.416+0000 7f1f865d6640 1 -- 172.21.15.155:0/2598893275 >> 172.21.15.155:0/2598893275 conn(0x7f1f800b7ea0 msgr2=0x7f1f800b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:08.419 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.416+0000 7f1f865d6640 1 -- 172.21.15.155:0/2598893275 shutdown_connections 2024-03-25T18:34:08.419 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.416+0000 7f1f865d6640 1 -- 172.21.15.155:0/2598893275 wait complete. 2024-03-25T18:34:08.419 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.417+0000 7f1f865d6640 10 monclient: build_initial_monmap 2024-03-25T18:34:08.419 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.417+0000 7f1f865d6640 10 monclient: monmap: 2024-03-25T18:34:08.419 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:34:08.419 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:34:08.419 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:34:08.419 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:34:08.419 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:08.419 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:34:08.419 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:34:08.420 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:34:08.420 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:34:08.420 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:34:08.420 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.418+0000 7f1f865d6640 1 librados: starting msgr at 2024-03-25T18:34:08.420 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.418+0000 7f1f865d6640 1 librados: starting objecter 2024-03-25T18:34:08.420 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.418+0000 7f1f865d6640 1 Processor -- start 2024-03-25T18:34:08.421 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.418+0000 7f1f865d6640 1 -- start start 2024-03-25T18:34:08.421 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.418+0000 7f1f865d6640 1 librados: setting wanted keys 2024-03-25T18:34:08.421 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.418+0000 7f1f865d6640 1 librados: calling monclient init 2024-03-25T18:34:08.421 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.418+0000 7f1f865d6640 10 monclient: init 2024-03-25T18:34:08.421 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.418+0000 7f1f865d6640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:08.421 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.418+0000 7f1f865d6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:34:08.421 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.419+0000 7f1f865d6640 10 monclient(hunting): picked mon.noname-c con 0x7f1f800a98f0 addr v1:172.21.15.155:6790/0 2024-03-25T18:34:08.421 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.419+0000 7f1f865d6640 10 monclient(hunting): picked mon.noname-a con 0x7f1f8013f710 addr v1:172.21.15.155:6789/0 2024-03-25T18:34:08.421 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.419+0000 7f1f865d6640 10 monclient(hunting): picked mon.noname-b con 0x7f1f801430c0 addr v1:172.21.15.167:6789/0 2024-03-25T18:34:08.421 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.419+0000 7f1f865d6640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f8012db40 con 0x7f1f8013f710 2024-03-25T18:34:08.421 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.419+0000 7f1f865d6640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f8013af60 con 0x7f1f801430c0 2024-03-25T18:34:08.421 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.419+0000 7f1f865d6640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f8013b240 con 0x7f1f800a98f0 2024-03-25T18:34:08.421 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.419+0000 7f1f865d6640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:08.421 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.419+0000 7f1f865d6640 10 monclient(hunting): authenticate will time out at 8094.728802s 2024-03-25T18:34:08.429 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.428+0000 7f1f7f7fe640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f1f8013f710 0x7f1f801df270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:45690/0 (socket says 172.21.15.155:45690) 2024-03-25T18:34:08.429 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.428+0000 7f1f7f7fe640 1 -- 172.21.15.155:0/204486731 learned_addr learned my addr 172.21.15.155:0/204486731 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:34:08.430 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.429+0000 7f1f7cff9640 10 client.?.objecter ms_handle_connect 0x7f1f8013f710 2024-03-25T18:34:08.430 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.429+0000 7f1f7cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:08.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.434+0000 7f1f7cff9640 1 -- 172.21.15.155:0/204486731 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 223070316 0 0) 0x7f1f8012db40 con 0x7f1f8013f710 2024-03-25T18:34:08.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.434+0000 7f1f7cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:34:08.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.434+0000 7f1f7cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:08.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.434+0000 7f1f7cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:08.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.434+0000 7f1f7cff9640 10 monclient(hunting): my global_id is 4650 2024-03-25T18:34:08.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.434+0000 7f1f7cff9640 1 -- 172.21.15.155:0/204486731 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f6c004eb0 con 0x7f1f8013f710 2024-03-25T18:34:08.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.437+0000 7f1f7cff9640 10 client.?.objecter ms_handle_connect 0x7f1f801430c0 2024-03-25T18:34:08.438 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.437+0000 7f1f7cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:08.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.439+0000 7f1f7cff9640 10 client.?.objecter ms_handle_connect 0x7f1f800a98f0 2024-03-25T18:34:08.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.439+0000 7f1f7cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:08.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.444+0000 7f1f7cff9640 1 -- 172.21.15.155:0/204486731 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4099338806 0 0) 0x7f1f6c004eb0 con 0x7f1f8013f710 2024-03-25T18:34:08.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.444+0000 7f1f7cff9640 1 -- 172.21.15.155:0/204486731 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1f8012db40 con 0x7f1f8013f710 2024-03-25T18:34:08.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.444+0000 7f1f7cff9640 1 -- 172.21.15.155:0/204486731 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f1f740019d0 con 0x7f1f8013f710 2024-03-25T18:34:08.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.444+0000 7f1f7cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:08.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.444+0000 7f1f7cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:34:08.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.444+0000 7f1f7cff9640 10 monclient(hunting): dump: 2024-03-25T18:34:08.446 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:34:08.446 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:34:08.446 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:08.446 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:08.446 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:08.446 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:34:08.446 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:34:08.446 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:34:08.446 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:34:08.447 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:34:08.447 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.444+0000 7f1f7cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:34:08.449 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.448+0000 7f1f7cff9640 1 -- 172.21.15.155:0/204486731 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2923749553 0 0) 0x7f1f8013af60 con 0x7f1f801430c0 2024-03-25T18:34:08.450 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.448+0000 7f1f7cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:34:08.450 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.448+0000 7f1f7cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:08.450 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.448+0000 7f1f7cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:08.450 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.448+0000 7f1f7cff9640 10 monclient(hunting): my global_id is 4648 2024-03-25T18:34:08.450 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.449+0000 7f1f7cff9640 1 -- 172.21.15.155:0/204486731 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f6c004eb0 con 0x7f1f801430c0 2024-03-25T18:34:08.451 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.450+0000 7f1f7cff9640 1 -- 172.21.15.155:0/204486731 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2486974238 0 0) 0x7f1f8013b240 con 0x7f1f800a98f0 2024-03-25T18:34:08.451 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.450+0000 7f1f7cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:34:08.451 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.450+0000 7f1f7cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:08.451 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.450+0000 7f1f7cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:08.451 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.450+0000 7f1f7cff9640 10 monclient(hunting): my global_id is 4661 2024-03-25T18:34:08.451 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.450+0000 7f1f7cff9640 1 -- 172.21.15.155:0/204486731 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f8013af60 con 0x7f1f800a98f0 2024-03-25T18:34:08.461 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.460+0000 7f1f7cff9640 1 -- 172.21.15.155:0/204486731 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 149654224 0 0) 0x7f1f8012db40 con 0x7f1f8013f710 2024-03-25T18:34:08.461 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.460+0000 7f1f7cff9640 1 -- 172.21.15.155:0/204486731 >> v1:172.21.15.155:6790/0 conn(0x7f1f800a98f0 legacy=0x7f1f80139d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:08.461 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.460+0000 7f1f7cff9640 1 -- 172.21.15.155:0/204486731 >> v1:172.21.15.167:6789/0 conn(0x7f1f801430c0 legacy=0x7f1f801e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:08.461 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.460+0000 7f1f7cff9640 10 monclient: _finish_hunting 0 2024-03-25T18:34:08.462 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.460+0000 7f1f7cff9640 1 monclient: found mon.a 2024-03-25T18:34:08.462 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.460+0000 7f1f7cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:08.462 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.460+0000 7f1f7cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:34:08.462 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.460+0000 7f1f7cff9640 1 -- 172.21.15.155:0/204486731 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f8013b520 con 0x7f1f8013f710 2024-03-25T18:34:08.462 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.460+0000 7f1f7cff9640 10 monclient: _finish_auth 0 2024-03-25T18:34:08.462 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.460+0000 7f1f7cff9640 10 monclient: _check_auth_tickets 2024-03-25T18:34:08.462 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.460+0000 7f1f7cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:08.462 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.460+0000 7f1f865d6640 5 monclient: authenticate success, global_id 4650 2024-03-25T18:34:08.462 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.460+0000 7f1f865d6640 10 monclient: _renew_subs 2024-03-25T18:34:08.462 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.460+0000 7f1f865d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:34:08.462 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.460+0000 7f1f865d6640 1 -- 172.21.15.155:0/204486731 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f80139900 con 0x7f1f8013f710 2024-03-25T18:34:08.462 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.460+0000 7f1f865d6640 10 client.4650.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:34:08.462 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.460+0000 7f1f865d6640 10 monclient: _renew_subs 2024-03-25T18:34:08.462 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.460+0000 7f1f865d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:34:08.462 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.460+0000 7f1f865d6640 1 -- 172.21.15.155:0/204486731 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f800b7240 con 0x7f1f8013f710 2024-03-25T18:34:08.463 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.460+0000 7f1f865d6640 1 librados: init done 2024-03-25T18:34:08.463 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.461+0000 7f1f865d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:08.463 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.461+0000 7f1f865d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:08.463 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.461+0000 7f1f865d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:34:08.463 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.461+0000 7f1f865d6640 1 -- 172.21.15.155:0/204486731 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f800627b0 con 0x7f1f8013f710 2024-03-25T18:34:08.756 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.754+0000 7f1f7cff9640 1 -- 172.21.15.155:0/204486731 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1f74001c50 con 0x7f1f8013f710 2024-03-25T18:34:08.756 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.754+0000 7f1f7cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:08.756 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.754+0000 7f1f7cff9640 1 -- 172.21.15.155:0/204486731 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f1f74003a60 con 0x7f1f8013f710 2024-03-25T18:34:08.756 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.754+0000 7f1f7cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:08.756 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.754+0000 7f1f7cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:34:08.756 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.754+0000 7f1f7cff9640 10 monclient: dump: 2024-03-25T18:34:08.756 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:34:08.756 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:34:08.756 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:08.756 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:08.757 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:08.757 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:34:08.757 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:34:08.757 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:34:08.757 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:34:08.757 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:34:08.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.754+0000 7f1f7e7fc640 4 set_mon_vals no callback set 2024-03-25T18:34:08.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.755+0000 7f1f7cff9640 1 -- 172.21.15.155:0/204486731 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7f1f74003c80 con 0x7f1f8013f710 2024-03-25T18:34:08.758 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.756+0000 7f1f7ffff640 10 monclient: get_auth_request con 0x7f1f6c06de60 auth_method 0 2024-03-25T18:34:08.758 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.756+0000 7f1f7cff9640 10 client.4650.objecter ms_handle_connect 0x7f1f6c06de60 2024-03-25T18:34:08.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.763+0000 7f1f7cff9640 1 -- 172.21.15.155:0/204486731 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7f1f74054820 con 0x7f1f8013f710 2024-03-25T18:34:08.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.763+0000 7f1f7cff9640 10 client.4650.objecter ms_dispatch 0x7f1f80000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:34:08.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.763+0000 7f1f7cff9640 3 client.4650.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:34:08.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.763+0000 7f1f7cff9640 3 client.4650.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:34:08.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.763+0000 7f1f7cff9640 20 client.4650.objecter dump_active .. 0 homeless 2024-03-25T18:34:08.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.763+0000 7f1f7cff9640 1 -- 172.21.15.155:0/204486731 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1f74050330 con 0x7f1f8013f710 2024-03-25T18:34:08.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.763+0000 7f1f7cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:08.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:08.763+0000 7f1f7cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:34:09.015 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:09.013+0000 7f1f865d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:09.015 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:09.013+0000 7f1f865d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:09.015 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:09.013+0000 7f1f865d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:34:09.015 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:09.013+0000 7f1f865d6640 1 -- 172.21.15.155:0/204486731 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1f8013d5f0 con 0x7f1f8013f710 2024-03-25T18:34:09.044 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:09.043+0000 7f1f7cff9640 1 -- 172.21.15.155:0/204486731 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7f1f74052260 con 0x7f1f8013f710 2024-03-25T18:34:09.045 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:09.043+0000 7f1f7cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:09.045 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:09.043+0000 7f1f7cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:34:09.045 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:34:09.045 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:34:09.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:09.046+0000 7f1f865d6640 10 librados: watch_flush enter 2024-03-25T18:34:09.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:09.046+0000 7f1f865d6640 10 librados: watch_flush exit 2024-03-25T18:34:09.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:09.046+0000 7f1f865d6640 20 client.4650.objecter shutdown clearing up homeless session... 2024-03-25T18:34:09.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:09.046+0000 7f1f865d6640 10 client.4650.objecter successfully canceled tick 2024-03-25T18:34:09.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:09.046+0000 7f1f865d6640 1 -- 172.21.15.155:0/204486731 >> v1:172.21.15.167:6800/2919070503 conn(0x7f1f6c06de60 legacy=0x7f1f6c070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:09.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:09.046+0000 7f1f865d6640 10 monclient: shutdown 2024-03-25T18:34:09.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:09.046+0000 7f1f865d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:09.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:09.046+0000 7f1f865d6640 1 -- 172.21.15.155:0/204486731 >> v1:172.21.15.155:6789/0 conn(0x7f1f8013f710 legacy=0x7f1f801df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:09.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:09.046+0000 7f1f865d6640 1 -- 172.21.15.155:0/204486731 shutdown_connections 2024-03-25T18:34:09.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:09.046+0000 7f1f865d6640 1 -- 172.21.15.155:0/204486731 >> 172.21.15.155:0/204486731 conn(0x7f1f800b7ea0 msgr2=0x7f1f80074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:09.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:09.046+0000 7f1f865d6640 1 -- 172.21.15.155:0/204486731 shutdown_connections 2024-03-25T18:34:09.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:09.047+0000 7f1f865d6640 1 -- 172.21.15.155:0/204486731 wait complete. 2024-03-25T18:34:09.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:09.047+0000 7f1f865d6640 1 librados: shutdown 2024-03-25T18:34:09.059 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:34:12.402 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:09:46.006 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:12.407 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:09:46.011 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:12.442 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:09:46.047 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:12.469 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:09:46.047 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:12.472 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:09:46.051 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:12.473 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:09:46.078 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:12.480 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:09:46.085 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:12.508 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:09:46.113 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:12.509 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:09:46.088 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:15.865 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:34:15.981 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:15.980+0000 7fd315ae4640 10 monclient: get_monmap_and_config 2024-03-25T18:34:15.981 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:15.980+0000 7fd315ae4640 10 monclient: build_initial_monmap 2024-03-25T18:34:15.981 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:15.980+0000 7fd315ae4640 10 monclient: monmap: 2024-03-25T18:34:15.982 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:34:15.982 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:34:15.982 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:34:15.981371+0000 2024-03-25T18:34:15.982 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:34:15.981371+0000 2024-03-25T18:34:15.982 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:15.982 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:34:15.982 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:34:15.982 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:34:15.982 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:34:15.982 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:34:15.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:15.981+0000 7fd315ae4640 1 Processor -- start 2024-03-25T18:34:15.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:15.981+0000 7fd315ae4640 1 -- start start 2024-03-25T18:34:15.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:15.981+0000 7fd315ae4640 10 monclient: init 2024-03-25T18:34:15.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:15.982+0000 7fd315ae4640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:15.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:15.982+0000 7fd315ae4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:34:15.984 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:15.982+0000 7fd315ae4640 10 monclient(hunting): picked mon.noname-b con 0x7fd3100b4d60 addr v1:172.21.15.155:6790/0 2024-03-25T18:34:15.984 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:15.982+0000 7fd315ae4640 10 monclient(hunting): picked mon.noname-a con 0x7fd3100b1730 addr v1:172.21.15.155:6789/0 2024-03-25T18:34:15.984 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:15.982+0000 7fd315ae4640 10 monclient(hunting): picked mon.noname-c con 0x7fd31014be90 addr v1:172.21.15.167:6789/0 2024-03-25T18:34:15.984 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:15.982+0000 7fd315ae4640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3100af940 con 0x7fd3100b1730 2024-03-25T18:34:15.984 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:15.982+0000 7fd315ae4640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3100b0a70 con 0x7fd31014be90 2024-03-25T18:34:15.984 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:15.982+0000 7fd315ae4640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3100b0d50 con 0x7fd3100b4d60 2024-03-25T18:34:15.984 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:15.982+0000 7fd315ae4640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:15.984 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:15.982+0000 7fd315ae4640 10 monclient(hunting): authenticate will time out at 8102.291904s 2024-03-25T18:34:15.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:15.985+0000 7fd30f7fe640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fd3100b4d60 0x7fd31013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:57602/0 (socket says 172.21.15.155:57602) 2024-03-25T18:34:15.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:15.985+0000 7fd30f7fe640 1 -- 172.21.15.155:0/3666646579 learned_addr learned my addr 172.21.15.155:0/3666646579 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:34:16.006 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.005+0000 7fd30e7fc640 1 -- 172.21.15.155:0/3666646579 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3635014150 0 0) 0x7fd3100af940 con 0x7fd3100b1730 2024-03-25T18:34:16.006 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.005+0000 7fd30e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:34:16.006 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.005+0000 7fd30e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:16.006 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.005+0000 7fd30e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:16.006 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.005+0000 7fd30e7fc640 10 monclient(hunting): my global_id is 4653 2024-03-25T18:34:16.006 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.005+0000 7fd30e7fc640 1 -- 172.21.15.155:0/3666646579 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2f8003520 con 0x7fd3100b1730 2024-03-25T18:34:16.011 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.011+0000 7fd30e7fc640 1 -- 172.21.15.155:0/3666646579 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1546725876 0 0) 0x7fd3100b0a70 con 0x7fd31014be90 2024-03-25T18:34:16.012 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.011+0000 7fd30e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:34:16.012 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.011+0000 7fd30e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:16.012 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.011+0000 7fd30e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:16.012 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.011+0000 7fd30e7fc640 10 monclient(hunting): my global_id is 4651 2024-03-25T18:34:16.012 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.011+0000 7fd30e7fc640 1 -- 172.21.15.155:0/3666646579 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3100af940 con 0x7fd31014be90 2024-03-25T18:34:16.015 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.013+0000 7fd30e7fc640 1 -- 172.21.15.155:0/3666646579 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 527161966 0 0) 0x7fd3100b0d50 con 0x7fd3100b4d60 2024-03-25T18:34:16.015 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.013+0000 7fd30e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:34:16.015 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.013+0000 7fd30e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:16.015 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.013+0000 7fd30e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:16.015 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.013+0000 7fd30e7fc640 10 monclient(hunting): my global_id is 4664 2024-03-25T18:34:16.015 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.013+0000 7fd30e7fc640 1 -- 172.21.15.155:0/3666646579 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3100b0a70 con 0x7fd3100b4d60 2024-03-25T18:34:16.029 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.027+0000 7fd30e7fc640 1 -- 172.21.15.155:0/3666646579 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 996948264 0 0) 0x7fd3100af940 con 0x7fd31014be90 2024-03-25T18:34:16.029 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.027+0000 7fd30e7fc640 1 -- 172.21.15.155:0/3666646579 >> v1:172.21.15.155:6790/0 conn(0x7fd3100b4d60 legacy=0x7fd31013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:16.029 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.027+0000 7fd30e7fc640 1 -- 172.21.15.155:0/3666646579 >> v1:172.21.15.155:6789/0 conn(0x7fd3100b1730 legacy=0x7fd3100b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:16.029 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.027+0000 7fd30e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:34:16.029 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.027+0000 7fd30e7fc640 1 monclient: found mon.noname-c 2024-03-25T18:34:16.029 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.027+0000 7fd30e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:16.029 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.027+0000 7fd30e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.167:6789/0 2024-03-25T18:34:16.029 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.027+0000 7fd30e7fc640 1 -- 172.21.15.155:0/3666646579 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3100b1030 con 0x7fd31014be90 2024-03-25T18:34:16.030 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.027+0000 7fd30e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:34:16.030 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.027+0000 7fd30e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:34:16.030 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.027+0000 7fd30e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:16.030 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.027+0000 7fd30e7fc640 1 -- 172.21.15.155:0/3666646579 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fd3080019f0 con 0x7fd31014be90 2024-03-25T18:34:16.030 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.027+0000 7fd30e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:16.030 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.027+0000 7fd30e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:34:16.030 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.027+0000 7fd30e7fc640 10 monclient: dump: 2024-03-25T18:34:16.030 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:34:16.030 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:34:16.030 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:16.030 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:16.030 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:16.030 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:34:16.030 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:34:16.031 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:34:16.031 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:34:16.031 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:34:16.031 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.027+0000 7fd315ae4640 5 monclient: authenticate success, global_id 4651 2024-03-25T18:34:16.031 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.027+0000 7fd315ae4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:16.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.032+0000 7fd30e7fc640 1 -- 172.21.15.155:0/3666646579 <== mon.1 v1:172.21.15.167:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd308001b50 con 0x7fd31014be90 2024-03-25T18:34:16.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.032+0000 7fd30e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:16.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.032+0000 7fd30e7fc640 1 -- 172.21.15.155:0/3666646579 <== mon.1 v1:172.21.15.167:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fd308001f60 con 0x7fd31014be90 2024-03-25T18:34:16.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.032+0000 7fd315ae4640 10 monclient: get_monmap_and_config success 2024-03-25T18:34:16.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.032+0000 7fd315ae4640 4 set_mon_vals no callback set 2024-03-25T18:34:16.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.032+0000 7fd30e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:16.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.032+0000 7fd30e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:16.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.032+0000 7fd30e7fc640 10 monclient: dump: 2024-03-25T18:34:16.034 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:34:16.034 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:34:16.034 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:16.035 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:16.035 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:16.035 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:34:16.035 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:34:16.035 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:34:16.035 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:34:16.035 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:34:16.037 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.033+0000 7fd315ae4640 10 monclient: shutdown 2024-03-25T18:34:16.037 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.033+0000 7fd315ae4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:16.037 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.033+0000 7fd315ae4640 1 -- 172.21.15.155:0/3666646579 >> v1:172.21.15.167:6789/0 conn(0x7fd31014be90 legacy=0x7fd31014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:16.037 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.033+0000 7fd315ae4640 1 -- 172.21.15.155:0/3666646579 shutdown_connections 2024-03-25T18:34:16.037 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.033+0000 7fd315ae4640 1 -- 172.21.15.155:0/3666646579 >> 172.21.15.155:0/3666646579 conn(0x7fd3100617a0 msgr2=0x7fd310061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:16.037 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.033+0000 7fd315ae4640 1 -- 172.21.15.155:0/3666646579 shutdown_connections 2024-03-25T18:34:16.038 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.033+0000 7fd315ae4640 1 -- 172.21.15.155:0/3666646579 wait complete. 2024-03-25T18:34:16.038 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.034+0000 7fd315ae4640 10 monclient: build_initial_monmap 2024-03-25T18:34:16.038 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.034+0000 7fd315ae4640 10 monclient: monmap: 2024-03-25T18:34:16.038 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:34:16.038 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:34:16.038 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:34:16.038 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:34:16.038 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:16.038 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:34:16.038 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:34:16.038 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:34:16.038 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:34:16.038 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:34:16.039 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.035+0000 7fd315ae4640 1 librados: starting msgr at 2024-03-25T18:34:16.039 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.035+0000 7fd315ae4640 1 librados: starting objecter 2024-03-25T18:34:16.039 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.035+0000 7fd315ae4640 1 Processor -- start 2024-03-25T18:34:16.039 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.035+0000 7fd315ae4640 1 -- start start 2024-03-25T18:34:16.039 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.035+0000 7fd315ae4640 1 librados: setting wanted keys 2024-03-25T18:34:16.039 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.035+0000 7fd315ae4640 1 librados: calling monclient init 2024-03-25T18:34:16.039 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.035+0000 7fd315ae4640 10 monclient: init 2024-03-25T18:34:16.039 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.036+0000 7fd315ae4640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:16.039 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.036+0000 7fd315ae4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:34:16.039 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.036+0000 7fd315ae4640 10 monclient(hunting): picked mon.noname-c con 0x7fd3100b1730 addr v1:172.21.15.155:6790/0 2024-03-25T18:34:16.039 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.036+0000 7fd315ae4640 10 monclient(hunting): picked mon.noname-b con 0x7fd3100b4d60 addr v1:172.21.15.167:6789/0 2024-03-25T18:34:16.039 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.036+0000 7fd315ae4640 10 monclient(hunting): picked mon.noname-a con 0x7fd31014be90 addr v1:172.21.15.155:6789/0 2024-03-25T18:34:16.039 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.036+0000 7fd315ae4640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd310142cc0 con 0x7fd31014be90 2024-03-25T18:34:16.039 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.036+0000 7fd315ae4640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd310142fa0 con 0x7fd3100b4d60 2024-03-25T18:34:16.040 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.036+0000 7fd315ae4640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3101db6a0 con 0x7fd3100b1730 2024-03-25T18:34:16.040 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.036+0000 7fd315ae4640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:16.040 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.036+0000 7fd315ae4640 10 monclient(hunting): authenticate will time out at 8102.345932s 2024-03-25T18:34:16.045 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.043+0000 7fd30f7fe640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fd3100b1730 0x7fd3101dacb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:57660/0 (socket says 172.21.15.155:57660) 2024-03-25T18:34:16.045 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.043+0000 7fd30f7fe640 1 -- 172.21.15.155:0/1570326854 learned_addr learned my addr 172.21.15.155:0/1570326854 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:34:16.045 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.043+0000 7fd2f7fff640 10 client.?.objecter ms_handle_connect 0x7fd3100b1730 2024-03-25T18:34:16.045 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.043+0000 7fd2f7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:16.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.059+0000 7fd2f7fff640 1 -- 172.21.15.155:0/1570326854 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3275639220 0 0) 0x7fd3101db6a0 con 0x7fd3100b1730 2024-03-25T18:34:16.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.060+0000 7fd2f7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:34:16.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.060+0000 7fd2f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:16.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.060+0000 7fd2f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:16.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.060+0000 7fd2f7fff640 10 monclient(hunting): my global_id is 4667 2024-03-25T18:34:16.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.060+0000 7fd2f7fff640 1 -- 172.21.15.155:0/1570326854 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2fc004eb0 con 0x7fd3100b1730 2024-03-25T18:34:16.064 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.063+0000 7fd2f7fff640 10 client.?.objecter ms_handle_connect 0x7fd3100b4d60 2024-03-25T18:34:16.064 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.063+0000 7fd2f7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:16.064 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.063+0000 7fd2f7fff640 10 client.?.objecter ms_handle_connect 0x7fd31014be90 2024-03-25T18:34:16.065 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.063+0000 7fd2f7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:16.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.075+0000 7fd2f7fff640 1 -- 172.21.15.155:0/1570326854 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1377920723 0 0) 0x7fd310142fa0 con 0x7fd3100b4d60 2024-03-25T18:34:16.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.075+0000 7fd2f7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:34:16.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.075+0000 7fd2f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:16.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.075+0000 7fd2f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:16.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.075+0000 7fd2f7fff640 10 monclient(hunting): my global_id is 4654 2024-03-25T18:34:16.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.075+0000 7fd2f7fff640 1 -- 172.21.15.155:0/1570326854 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3101db6a0 con 0x7fd3100b4d60 2024-03-25T18:34:16.078 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.077+0000 7fd2f7fff640 1 -- 172.21.15.155:0/1570326854 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2173157704 0 0) 0x7fd310142cc0 con 0x7fd31014be90 2024-03-25T18:34:16.078 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.077+0000 7fd2f7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:34:16.078 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.077+0000 7fd2f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:16.078 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.077+0000 7fd2f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:16.078 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.077+0000 7fd2f7fff640 10 monclient(hunting): my global_id is 4656 2024-03-25T18:34:16.078 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.077+0000 7fd2f7fff640 1 -- 172.21.15.155:0/1570326854 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd310142fa0 con 0x7fd31014be90 2024-03-25T18:34:16.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.092+0000 7fd2f7fff640 1 -- 172.21.15.155:0/1570326854 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1123257331 0 0) 0x7fd3101db6a0 con 0x7fd3100b4d60 2024-03-25T18:34:16.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.092+0000 7fd2f7fff640 1 -- 172.21.15.155:0/1570326854 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd310142cc0 con 0x7fd3100b4d60 2024-03-25T18:34:16.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.092+0000 7fd2f7fff640 1 -- 172.21.15.155:0/1570326854 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fd304001950 con 0x7fd3100b4d60 2024-03-25T18:34:16.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.092+0000 7fd2f7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:16.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.093+0000 7fd2f7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:34:16.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.093+0000 7fd2f7fff640 10 monclient(hunting): dump: 2024-03-25T18:34:16.094 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:34:16.094 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:34:16.094 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:16.094 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:16.094 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:16.095 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:34:16.095 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:34:16.095 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:34:16.095 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:34:16.095 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:34:16.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.093+0000 7fd2f7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:34:16.097 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.096+0000 7fd2f7fff640 1 -- 172.21.15.155:0/1570326854 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4273445405 0 0) 0x7fd2fc004eb0 con 0x7fd3100b1730 2024-03-25T18:34:16.097 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.096+0000 7fd2f7fff640 1 -- 172.21.15.155:0/1570326854 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd3101db6a0 con 0x7fd3100b1730 2024-03-25T18:34:16.097 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.096+0000 7fd2f7fff640 1 -- 172.21.15.155:0/1570326854 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fd300001910 con 0x7fd3100b1730 2024-03-25T18:34:16.097 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.096+0000 7fd2f7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:16.098 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.096+0000 7fd2f7fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:34:16.098 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.096+0000 7fd2f7fff640 10 monclient(hunting): dump: 2024-03-25T18:34:16.098 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:34:16.098 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:34:16.098 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:16.098 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:16.098 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:16.098 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:34:16.098 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:34:16.098 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:34:16.098 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:34:16.098 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:34:16.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.102+0000 7fd2f7fff640 1 -- 172.21.15.155:0/1570326854 <== mon.1 v1:172.21.15.167:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1878933271 0 0) 0x7fd310142cc0 con 0x7fd3100b4d60 2024-03-25T18:34:16.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.102+0000 7fd2f7fff640 1 -- 172.21.15.155:0/1570326854 >> v1:172.21.15.155:6790/0 conn(0x7fd3100b1730 legacy=0x7fd3101dacb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:16.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.102+0000 7fd2f7fff640 1 -- 172.21.15.155:0/1570326854 >> v1:172.21.15.155:6789/0 conn(0x7fd31014be90 legacy=0x7fd310141090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:16.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.102+0000 7fd2f7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:34:16.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.102+0000 7fd2f7fff640 1 monclient: found mon.b 2024-03-25T18:34:16.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.102+0000 7fd2f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:16.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.102+0000 7fd2f7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:34:16.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.102+0000 7fd2f7fff640 1 -- 172.21.15.155:0/1570326854 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3101db3c0 con 0x7fd3100b4d60 2024-03-25T18:34:16.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.102+0000 7fd2f7fff640 10 monclient: _finish_auth 0 2024-03-25T18:34:16.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.102+0000 7fd2f7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:34:16.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.102+0000 7fd2f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:16.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.102+0000 7fd315ae4640 5 monclient: authenticate success, global_id 4654 2024-03-25T18:34:16.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.102+0000 7fd315ae4640 10 monclient: _renew_subs 2024-03-25T18:34:16.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.102+0000 7fd315ae4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:34:16.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.102+0000 7fd315ae4640 1 -- 172.21.15.155:0/1570326854 --> v1:172.21.15.167:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3101d5000 con 0x7fd3100b4d60 2024-03-25T18:34:16.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.102+0000 7fd315ae4640 10 client.4654.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:34:16.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.102+0000 7fd315ae4640 10 monclient: _renew_subs 2024-03-25T18:34:16.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.102+0000 7fd315ae4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:34:16.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.102+0000 7fd315ae4640 1 -- 172.21.15.155:0/1570326854 --> v1:172.21.15.167:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd310133680 con 0x7fd3100b4d60 2024-03-25T18:34:16.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.102+0000 7fd315ae4640 1 librados: init done 2024-03-25T18:34:16.106 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.103+0000 7fd315ae4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:16.106 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.103+0000 7fd315ae4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:16.106 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.103+0000 7fd315ae4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:34:16.106 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.103+0000 7fd315ae4640 1 -- 172.21.15.155:0/1570326854 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3101db830 con 0x7fd3100b4d60 2024-03-25T18:34:16.382 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.381+0000 7fd2f7fff640 1 -- 172.21.15.155:0/1570326854 <== mon.1 v1:172.21.15.167:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd304001d60 con 0x7fd3100b4d60 2024-03-25T18:34:16.382 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.381+0000 7fd2f7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:16.382 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.381+0000 7fd2f7fff640 1 -- 172.21.15.155:0/1570326854 <== mon.1 v1:172.21.15.167:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fd3040037b0 con 0x7fd3100b4d60 2024-03-25T18:34:16.382 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.381+0000 7fd2f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:16.382 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.381+0000 7fd2f7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:16.382 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.381+0000 7fd2f7fff640 10 monclient: dump: 2024-03-25T18:34:16.382 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:34:16.382 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:34:16.383 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:16.383 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:16.383 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:16.383 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:34:16.383 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:34:16.383 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:34:16.383 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:34:16.383 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:34:16.383 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.381+0000 7fd30dffb640 4 set_mon_vals no callback set 2024-03-25T18:34:16.383 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.382+0000 7fd2f7fff640 1 -- 172.21.15.155:0/1570326854 <== mon.1 v1:172.21.15.167:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7fd3040039d0 con 0x7fd3100b4d60 2024-03-25T18:34:16.384 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.383+0000 7fd30f7fe640 10 monclient: get_auth_request con 0x7fd2fc06dec0 auth_method 0 2024-03-25T18:34:16.384 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.383+0000 7fd2f7fff640 10 client.4654.objecter ms_handle_connect 0x7fd2fc06dec0 2024-03-25T18:34:16.391 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.390+0000 7fd2f7fff640 1 -- 172.21.15.155:0/1570326854 <== mon.1 v1:172.21.15.167:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7fd304053530 con 0x7fd3100b4d60 2024-03-25T18:34:16.391 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.390+0000 7fd2f7fff640 10 client.4654.objecter ms_dispatch 0x7fd310000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:34:16.391 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.390+0000 7fd2f7fff640 3 client.4654.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:34:16.391 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.390+0000 7fd2f7fff640 3 client.4654.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:34:16.391 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.390+0000 7fd2f7fff640 20 client.4654.objecter dump_active .. 0 homeless 2024-03-25T18:34:16.392 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.391+0000 7fd2f7fff640 1 -- 172.21.15.155:0/1570326854 <== mon.1 v1:172.21.15.167:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd30404f040 con 0x7fd3100b4d60 2024-03-25T18:34:16.392 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.391+0000 7fd2f7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:16.392 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.391+0000 7fd2f7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:34:16.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.621+0000 7fd315ae4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:16.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.621+0000 7fd315ae4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:16.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.621+0000 7fd315ae4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:34:16.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.621+0000 7fd315ae4640 1 -- 172.21.15.155:0/1570326854 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd3100627b0 con 0x7fd3100b4d60 2024-03-25T18:34:16.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.660+0000 7fd2f7fff640 1 -- 172.21.15.155:0/1570326854 <== mon.1 v1:172.21.15.167:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7fd304050f70 con 0x7fd3100b4d60 2024-03-25T18:34:16.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.660+0000 7fd2f7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:16.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.660+0000 7fd2f7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:34:16.662 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:34:16.662 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:34:16.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.662+0000 7fd315ae4640 10 librados: watch_flush enter 2024-03-25T18:34:16.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.662+0000 7fd315ae4640 10 librados: watch_flush exit 2024-03-25T18:34:16.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.662+0000 7fd315ae4640 20 client.4654.objecter shutdown clearing up homeless session... 2024-03-25T18:34:16.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.662+0000 7fd315ae4640 10 client.4654.objecter successfully canceled tick 2024-03-25T18:34:16.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.662+0000 7fd315ae4640 1 -- 172.21.15.155:0/1570326854 >> v1:172.21.15.167:6800/2919070503 conn(0x7fd2fc06dec0 legacy=0x7fd2fc070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:16.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.662+0000 7fd315ae4640 10 monclient: shutdown 2024-03-25T18:34:16.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.662+0000 7fd315ae4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:16.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.662+0000 7fd315ae4640 1 -- 172.21.15.155:0/1570326854 >> v1:172.21.15.167:6789/0 conn(0x7fd3100b4d60 legacy=0x7fd310132d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:16.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.662+0000 7fd315ae4640 1 -- 172.21.15.155:0/1570326854 shutdown_connections 2024-03-25T18:34:16.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.662+0000 7fd315ae4640 1 -- 172.21.15.155:0/1570326854 >> 172.21.15.155:0/1570326854 conn(0x7fd3100617a0 msgr2=0x7fd3100b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:16.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.663+0000 7fd315ae4640 1 -- 172.21.15.155:0/1570326854 shutdown_connections 2024-03-25T18:34:16.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.663+0000 7fd315ae4640 1 -- 172.21.15.155:0/1570326854 wait complete. 2024-03-25T18:34:16.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:16.663+0000 7fd315ae4640 1 librados: shutdown 2024-03-25T18:34:16.676 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:34:23.483 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:34:23.601 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.600+0000 7f37f1c63640 10 monclient: get_monmap_and_config 2024-03-25T18:34:23.602 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.600+0000 7f37f1c63640 10 monclient: build_initial_monmap 2024-03-25T18:34:23.602 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.600+0000 7f37f1c63640 10 monclient: monmap: 2024-03-25T18:34:23.602 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:34:23.602 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:34:23.602 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:34:23.601291+0000 2024-03-25T18:34:23.602 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:34:23.601291+0000 2024-03-25T18:34:23.602 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:23.602 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:34:23.602 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:34:23.602 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:34:23.602 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:34:23.602 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:34:23.604 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.601+0000 7f37f1c63640 1 Processor -- start 2024-03-25T18:34:23.604 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.601+0000 7f37f1c63640 1 -- start start 2024-03-25T18:34:23.604 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.601+0000 7f37f1c63640 10 monclient: init 2024-03-25T18:34:23.604 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.602+0000 7f37f1c63640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:23.604 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.602+0000 7f37f1c63640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:34:23.604 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.602+0000 7f37f1c63640 10 monclient(hunting): picked mon.noname-b con 0x7f37ec0a98f0 addr v1:172.21.15.155:6790/0 2024-03-25T18:34:23.604 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.602+0000 7f37f1c63640 10 monclient(hunting): picked mon.noname-c con 0x7f37ec13f710 addr v1:172.21.15.167:6789/0 2024-03-25T18:34:23.604 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.602+0000 7f37f1c63640 10 monclient(hunting): picked mon.noname-a con 0x7f37ec1430c0 addr v1:172.21.15.155:6789/0 2024-03-25T18:34:23.604 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.602+0000 7f37f1c63640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37ec142520 con 0x7f37ec1430c0 2024-03-25T18:34:23.605 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.602+0000 7f37f1c63640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37ec0b28a0 con 0x7f37ec13f710 2024-03-25T18:34:23.605 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.602+0000 7f37f1c63640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37ec0b2a30 con 0x7f37ec0a98f0 2024-03-25T18:34:23.605 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.602+0000 7f37f1c63640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:23.605 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.602+0000 7f37f1c63640 10 monclient(hunting): authenticate will time out at 8109.912221s 2024-03-25T18:34:23.614 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.613+0000 7f37eaffd640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f37ec13f710 0x7f37ec1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:54202/0 (socket says 172.21.15.155:54202) 2024-03-25T18:34:23.614 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.613+0000 7f37eaffd640 1 -- 172.21.15.155:0/2287159522 learned_addr learned my addr 172.21.15.155:0/2287159522 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:34:23.626 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.625+0000 7f37ea7fc640 1 -- 172.21.15.155:0/2287159522 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2365767727 0 0) 0x7f37ec142520 con 0x7f37ec1430c0 2024-03-25T18:34:23.626 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.625+0000 7f37ea7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:34:23.627 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.625+0000 7f37ea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:23.627 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.625+0000 7f37ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:23.627 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.625+0000 7f37ea7fc640 10 monclient(hunting): my global_id is 4659 2024-03-25T18:34:23.627 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.625+0000 7f37ea7fc640 1 -- 172.21.15.155:0/2287159522 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37d4003520 con 0x7f37ec1430c0 2024-03-25T18:34:23.631 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.630+0000 7f37ea7fc640 1 -- 172.21.15.155:0/2287159522 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2283892864 0 0) 0x7f37ec0b2a30 con 0x7f37ec0a98f0 2024-03-25T18:34:23.631 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.630+0000 7f37ea7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:34:23.631 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.630+0000 7f37ea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:23.631 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.630+0000 7f37ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:23.632 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.630+0000 7f37ea7fc640 10 monclient(hunting): my global_id is 4670 2024-03-25T18:34:23.632 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.630+0000 7f37ea7fc640 1 -- 172.21.15.155:0/2287159522 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37ec142520 con 0x7f37ec0a98f0 2024-03-25T18:34:23.632 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.630+0000 7f37ea7fc640 1 -- 172.21.15.155:0/2287159522 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 724983820 0 0) 0x7f37ec0b28a0 con 0x7f37ec13f710 2024-03-25T18:34:23.632 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.630+0000 7f37ea7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:34:23.632 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.630+0000 7f37ea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:23.632 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.630+0000 7f37ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:23.632 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.630+0000 7f37ea7fc640 10 monclient(hunting): my global_id is 4657 2024-03-25T18:34:23.632 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.630+0000 7f37ea7fc640 1 -- 172.21.15.155:0/2287159522 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37ec0b2a30 con 0x7f37ec13f710 2024-03-25T18:34:23.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.648+0000 7f37ea7fc640 1 -- 172.21.15.155:0/2287159522 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 419794491 0 0) 0x7f37ec0b2a30 con 0x7f37ec13f710 2024-03-25T18:34:23.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.648+0000 7f37ea7fc640 1 -- 172.21.15.155:0/2287159522 >> v1:172.21.15.155:6790/0 conn(0x7f37ec0a98f0 legacy=0x7f37ec0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:23.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.648+0000 7f37ea7fc640 1 -- 172.21.15.155:0/2287159522 >> v1:172.21.15.155:6789/0 conn(0x7f37ec1430c0 legacy=0x7f37ec14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:23.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.648+0000 7f37ea7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:34:23.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.648+0000 7f37ea7fc640 1 monclient: found mon.noname-c 2024-03-25T18:34:23.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.648+0000 7f37ea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:23.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.648+0000 7f37ea7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.167:6789/0 2024-03-25T18:34:23.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.648+0000 7f37ea7fc640 1 -- 172.21.15.155:0/2287159522 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37ec0b44c0 con 0x7f37ec13f710 2024-03-25T18:34:23.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.648+0000 7f37ea7fc640 10 monclient: _finish_auth 0 2024-03-25T18:34:23.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.648+0000 7f37ea7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:34:23.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.648+0000 7f37ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:23.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.648+0000 7f37ea7fc640 1 -- 172.21.15.155:0/2287159522 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f37dc0015b0 con 0x7f37ec13f710 2024-03-25T18:34:23.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.648+0000 7f37ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:23.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.648+0000 7f37ea7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:34:23.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.648+0000 7f37ea7fc640 10 monclient: dump: 2024-03-25T18:34:23.651 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:34:23.651 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:34:23.651 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:23.651 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:23.651 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:23.651 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:34:23.651 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:34:23.651 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:34:23.651 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:34:23.651 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:34:23.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.648+0000 7f37f1c63640 5 monclient: authenticate success, global_id 4657 2024-03-25T18:34:23.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.648+0000 7f37f1c63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:23.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.665+0000 7f37ea7fc640 1 -- 172.21.15.155:0/2287159522 <== mon.1 v1:172.21.15.167:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f37dc001710 con 0x7f37ec13f710 2024-03-25T18:34:23.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.665+0000 7f37ea7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:23.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.665+0000 7f37ea7fc640 1 -- 172.21.15.155:0/2287159522 <== mon.1 v1:172.21.15.167:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f37dc001b60 con 0x7f37ec13f710 2024-03-25T18:34:23.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.665+0000 7f37ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:23.667 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.665+0000 7f37ea7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:23.667 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.665+0000 7f37ea7fc640 10 monclient: dump: 2024-03-25T18:34:23.667 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:34:23.667 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:34:23.667 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:23.667 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:23.667 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:23.667 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:34:23.667 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:34:23.667 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:34:23.667 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:34:23.667 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:34:23.667 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.665+0000 7f37f1c63640 10 monclient: get_monmap_and_config success 2024-03-25T18:34:23.667 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.665+0000 7f37f1c63640 4 set_mon_vals no callback set 2024-03-25T18:34:23.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.666+0000 7f37f1c63640 10 monclient: shutdown 2024-03-25T18:34:23.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.666+0000 7f37f1c63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:23.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.666+0000 7f37f1c63640 1 -- 172.21.15.155:0/2287159522 >> v1:172.21.15.167:6789/0 conn(0x7f37ec13f710 legacy=0x7f37ec1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:23.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.666+0000 7f37f1c63640 1 -- 172.21.15.155:0/2287159522 shutdown_connections 2024-03-25T18:34:23.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.666+0000 7f37f1c63640 1 -- 172.21.15.155:0/2287159522 >> 172.21.15.155:0/2287159522 conn(0x7f37ec0b7ea0 msgr2=0x7f37ec0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:23.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.666+0000 7f37f1c63640 1 -- 172.21.15.155:0/2287159522 shutdown_connections 2024-03-25T18:34:23.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.666+0000 7f37f1c63640 1 -- 172.21.15.155:0/2287159522 wait complete. 2024-03-25T18:34:23.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.667+0000 7f37f1c63640 10 monclient: build_initial_monmap 2024-03-25T18:34:23.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.667+0000 7f37f1c63640 10 monclient: monmap: 2024-03-25T18:34:23.669 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:34:23.669 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:34:23.669 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:34:23.669 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:34:23.669 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:23.669 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:34:23.669 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:34:23.670 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:34:23.670 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:34:23.670 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:34:23.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.668+0000 7f37f1c63640 1 librados: starting msgr at 2024-03-25T18:34:23.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.668+0000 7f37f1c63640 1 librados: starting objecter 2024-03-25T18:34:23.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.668+0000 7f37f1c63640 1 Processor -- start 2024-03-25T18:34:23.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.668+0000 7f37f1c63640 1 -- start start 2024-03-25T18:34:23.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.668+0000 7f37f1c63640 1 librados: setting wanted keys 2024-03-25T18:34:23.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.668+0000 7f37f1c63640 1 librados: calling monclient init 2024-03-25T18:34:23.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.668+0000 7f37f1c63640 10 monclient: init 2024-03-25T18:34:23.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.669+0000 7f37f1c63640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:23.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.669+0000 7f37f1c63640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:34:23.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.669+0000 7f37f1c63640 10 monclient(hunting): picked mon.noname-b con 0x7f37ec0a98f0 addr v1:172.21.15.167:6789/0 2024-03-25T18:34:23.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.669+0000 7f37f1c63640 10 monclient(hunting): picked mon.noname-c con 0x7f37ec13f710 addr v1:172.21.15.155:6790/0 2024-03-25T18:34:23.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.669+0000 7f37f1c63640 10 monclient(hunting): picked mon.noname-a con 0x7f37ec1430c0 addr v1:172.21.15.155:6789/0 2024-03-25T18:34:23.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.669+0000 7f37f1c63640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37ec13af10 con 0x7f37ec1430c0 2024-03-25T18:34:23.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.669+0000 7f37f1c63640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37ec13b1f0 con 0x7f37ec0a98f0 2024-03-25T18:34:23.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.669+0000 7f37f1c63640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37ec13b4d0 con 0x7f37ec13f710 2024-03-25T18:34:23.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.669+0000 7f37f1c63640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:23.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.669+0000 7f37f1c63640 10 monclient(hunting): authenticate will time out at 8109.979474s 2024-03-25T18:34:23.678 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.677+0000 7f37eaffd640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f37ec13f710 0x7f37ec13bbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:32808/0 (socket says 172.21.15.155:32808) 2024-03-25T18:34:23.678 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.677+0000 7f37eaffd640 1 -- 172.21.15.155:0/2717495002 learned_addr learned my addr 172.21.15.155:0/2717495002 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:34:23.680 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.679+0000 7f37d3fff640 10 client.?.objecter ms_handle_connect 0x7f37ec13f710 2024-03-25T18:34:23.681 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.679+0000 7f37d3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:23.687 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.686+0000 7f37d3fff640 10 client.?.objecter ms_handle_connect 0x7f37ec1430c0 2024-03-25T18:34:23.687 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.686+0000 7f37d3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:23.690 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.689+0000 7f37d3fff640 10 client.?.objecter ms_handle_connect 0x7f37ec0a98f0 2024-03-25T18:34:23.690 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.689+0000 7f37d3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:23.694 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.692+0000 7f37d3fff640 1 -- 172.21.15.155:0/2717495002 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 532009595 0 0) 0x7f37ec13b4d0 con 0x7f37ec13f710 2024-03-25T18:34:23.694 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.692+0000 7f37d3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:34:23.694 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.692+0000 7f37d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:23.694 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.692+0000 7f37d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:23.694 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.692+0000 7f37d3fff640 10 monclient(hunting): my global_id is 4673 2024-03-25T18:34:23.694 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.692+0000 7f37d3fff640 1 -- 172.21.15.155:0/2717495002 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37d8004eb0 con 0x7f37ec13f710 2024-03-25T18:34:23.697 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.696+0000 7f37d3fff640 1 -- 172.21.15.155:0/2717495002 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2130206270 0 0) 0x7f37ec13af10 con 0x7f37ec1430c0 2024-03-25T18:34:23.697 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.696+0000 7f37d3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:34:23.697 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.696+0000 7f37d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:23.697 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.696+0000 7f37d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:23.698 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.696+0000 7f37d3fff640 10 monclient(hunting): my global_id is 4662 2024-03-25T18:34:23.698 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.696+0000 7f37d3fff640 1 -- 172.21.15.155:0/2717495002 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37ec13b4d0 con 0x7f37ec1430c0 2024-03-25T18:34:23.705 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.704+0000 7f37d3fff640 1 -- 172.21.15.155:0/2717495002 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3366742098 0 0) 0x7f37ec13b1f0 con 0x7f37ec0a98f0 2024-03-25T18:34:23.705 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.704+0000 7f37d3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:34:23.705 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.704+0000 7f37d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:23.705 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.704+0000 7f37d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:23.705 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.704+0000 7f37d3fff640 10 monclient(hunting): my global_id is 4660 2024-03-25T18:34:23.705 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.704+0000 7f37d3fff640 1 -- 172.21.15.155:0/2717495002 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37ec13af10 con 0x7f37ec0a98f0 2024-03-25T18:34:23.711 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.710+0000 7f37d3fff640 1 -- 172.21.15.155:0/2717495002 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3825306829 0 0) 0x7f37d8004eb0 con 0x7f37ec13f710 2024-03-25T18:34:23.711 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.710+0000 7f37d3fff640 1 -- 172.21.15.155:0/2717495002 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f37ec13b1f0 con 0x7f37ec13f710 2024-03-25T18:34:23.711 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.710+0000 7f37d3fff640 1 -- 172.21.15.155:0/2717495002 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f37dc001ae0 con 0x7f37ec13f710 2024-03-25T18:34:23.711 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.710+0000 7f37d3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:23.711 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.710+0000 7f37d3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:34:23.711 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.710+0000 7f37d3fff640 10 monclient(hunting): dump: 2024-03-25T18:34:23.711 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:34:23.711 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:34:23.711 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:23.712 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:23.712 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:23.712 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:34:23.712 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:34:23.712 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:34:23.712 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:34:23.712 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:34:23.712 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.710+0000 7f37d3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:34:23.713 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.712+0000 7f37d3fff640 1 -- 172.21.15.155:0/2717495002 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1640683512 0 0) 0x7f37ec13b4d0 con 0x7f37ec1430c0 2024-03-25T18:34:23.713 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.712+0000 7f37d3fff640 1 -- 172.21.15.155:0/2717495002 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f37d8004eb0 con 0x7f37ec1430c0 2024-03-25T18:34:23.713 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.712+0000 7f37d3fff640 1 -- 172.21.15.155:0/2717495002 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f37e4001ab0 con 0x7f37ec1430c0 2024-03-25T18:34:23.713 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.712+0000 7f37d3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:23.713 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.712+0000 7f37d3fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:34:23.713 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.712+0000 7f37d3fff640 10 monclient(hunting): dump: 2024-03-25T18:34:23.713 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:34:23.713 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:34:23.713 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:23.713 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:23.713 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:23.714 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:34:23.714 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:34:23.714 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:34:23.714 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:34:23.714 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:34:23.716 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.715+0000 7f37d3fff640 1 -- 172.21.15.155:0/2717495002 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 934781690 0 0) 0x7f37ec13af10 con 0x7f37ec0a98f0 2024-03-25T18:34:23.716 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.715+0000 7f37d3fff640 1 -- 172.21.15.155:0/2717495002 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f37ec13b4d0 con 0x7f37ec0a98f0 2024-03-25T18:34:23.716 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.715+0000 7f37d3fff640 1 -- 172.21.15.155:0/2717495002 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f37e0001930 con 0x7f37ec0a98f0 2024-03-25T18:34:23.716 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.715+0000 7f37d3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:23.717 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.715+0000 7f37d3fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:23.717 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.715+0000 7f37d3fff640 10 monclient(hunting): dump: 2024-03-25T18:34:23.717 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:34:23.717 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:34:23.717 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:23.717 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:23.717 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:23.717 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:34:23.717 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:34:23.717 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:34:23.717 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:34:23.717 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:34:23.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.717+0000 7f37d3fff640 1 -- 172.21.15.155:0/2717495002 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2514854156 0 0) 0x7f37ec13b1f0 con 0x7f37ec13f710 2024-03-25T18:34:23.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.717+0000 7f37d3fff640 1 -- 172.21.15.155:0/2717495002 >> v1:172.21.15.167:6789/0 conn(0x7f37ec0a98f0 legacy=0x7f37ec139db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:23.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.717+0000 7f37d3fff640 1 -- 172.21.15.155:0/2717495002 >> v1:172.21.15.155:6789/0 conn(0x7f37ec1430c0 legacy=0x7f37ec13c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:23.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.717+0000 7f37d3fff640 10 monclient: _finish_hunting 0 2024-03-25T18:34:23.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.717+0000 7f37d3fff640 1 monclient: found mon.c 2024-03-25T18:34:23.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.717+0000 7f37d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:23.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.717+0000 7f37d3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:34:23.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.717+0000 7f37d3fff640 1 -- 172.21.15.155:0/2717495002 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37ec13b7b0 con 0x7f37ec13f710 2024-03-25T18:34:23.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.717+0000 7f37d3fff640 10 monclient: _finish_auth 0 2024-03-25T18:34:23.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.717+0000 7f37d3fff640 10 monclient: _check_auth_tickets 2024-03-25T18:34:23.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.717+0000 7f37d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:23.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.717+0000 7f37f1c63640 5 monclient: authenticate success, global_id 4673 2024-03-25T18:34:23.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.717+0000 7f37f1c63640 10 monclient: _renew_subs 2024-03-25T18:34:23.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.717+0000 7f37f1c63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:34:23.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.717+0000 7f37f1c63640 1 -- 172.21.15.155:0/2717495002 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37ec0b79d0 con 0x7f37ec13f710 2024-03-25T18:34:23.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.717+0000 7f37f1c63640 10 client.4673.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:34:23.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.717+0000 7f37f1c63640 10 monclient: _renew_subs 2024-03-25T18:34:23.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.717+0000 7f37f1c63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:34:23.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.717+0000 7f37f1c63640 1 -- 172.21.15.155:0/2717495002 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f37ec13c9f0 con 0x7f37ec13f710 2024-03-25T18:34:23.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.717+0000 7f37f1c63640 1 librados: init done 2024-03-25T18:34:23.721 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.718+0000 7f37f1c63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:23.721 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.718+0000 7f37f1c63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:23.721 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.718+0000 7f37f1c63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:34:23.721 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.718+0000 7f37f1c63640 1 -- 172.21.15.155:0/2717495002 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37ec074cc0 con 0x7f37ec13f710 2024-03-25T18:34:23.722 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.721+0000 7f37d3fff640 1 -- 172.21.15.155:0/2717495002 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f37dc004a90 con 0x7f37ec13f710 2024-03-25T18:34:23.722 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.721+0000 7f37d3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:23.722 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:23.721+0000 7f37e9ffb640 4 set_mon_vals no callback set 2024-03-25T18:34:24.018 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:24.016+0000 7f37d3fff640 1 -- 172.21.15.155:0/2717495002 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f37dc004da0 con 0x7f37ec13f710 2024-03-25T18:34:24.018 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:24.016+0000 7f37d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:24.018 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:24.016+0000 7f37d3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:34:24.019 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:24.016+0000 7f37d3fff640 10 monclient: dump: 2024-03-25T18:34:24.019 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:34:24.019 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:34:24.019 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:24.019 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:24.019 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:24.019 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:34:24.019 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:34:24.019 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:34:24.019 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:34:24.019 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:34:24.020 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:24.017+0000 7f37d3fff640 1 -- 172.21.15.155:0/2717495002 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7f37dc01ad70 con 0x7f37ec13f710 2024-03-25T18:34:24.020 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:24.017+0000 7f37eb7fe640 10 monclient: get_auth_request con 0x7f37d806e0f0 auth_method 0 2024-03-25T18:34:24.020 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:24.018+0000 7f37d3fff640 10 client.4673.objecter ms_handle_connect 0x7f37d806e0f0 2024-03-25T18:34:24.026 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:24.024+0000 7f37d3fff640 1 -- 172.21.15.155:0/2717495002 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7f37dc0549b0 con 0x7f37ec13f710 2024-03-25T18:34:24.026 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:24.024+0000 7f37d3fff640 10 client.4673.objecter ms_dispatch 0x7f37ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:34:24.026 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:24.024+0000 7f37d3fff640 3 client.4673.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:34:24.026 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:24.024+0000 7f37d3fff640 3 client.4673.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:34:24.026 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:24.024+0000 7f37d3fff640 20 client.4673.objecter dump_active .. 0 homeless 2024-03-25T18:34:24.026 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:24.024+0000 7f37d3fff640 1 -- 172.21.15.155:0/2717495002 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f37dc0502f0 con 0x7f37ec13f710 2024-03-25T18:34:24.026 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:24.024+0000 7f37d3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:24.026 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:24.024+0000 7f37d3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:34:24.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:24.272+0000 7f37f1c63640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:24.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:24.272+0000 7f37f1c63640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:24.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:24.272+0000 7f37f1c63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:34:24.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:24.272+0000 7f37f1c63640 1 -- 172.21.15.155:0/2717495002 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f37ec0b7b60 con 0x7f37ec13f710 2024-03-25T18:34:24.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:24.340+0000 7f37d3fff640 1 -- 172.21.15.155:0/2717495002 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7f37dc0523f0 con 0x7f37ec13f710 2024-03-25T18:34:24.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:24.340+0000 7f37d3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:24.343 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:24.340+0000 7f37d3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:34:24.343 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:34:24.343 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:34:24.345 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:24.342+0000 7f37f1c63640 10 librados: watch_flush enter 2024-03-25T18:34:24.345 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:24.342+0000 7f37f1c63640 10 librados: watch_flush exit 2024-03-25T18:34:24.345 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:24.342+0000 7f37f1c63640 20 client.4673.objecter shutdown clearing up homeless session... 2024-03-25T18:34:24.345 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:24.342+0000 7f37f1c63640 10 client.4673.objecter successfully canceled tick 2024-03-25T18:34:24.345 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:24.342+0000 7f37f1c63640 1 -- 172.21.15.155:0/2717495002 >> v1:172.21.15.167:6800/2919070503 conn(0x7f37d806e0f0 legacy=0x7f37d80705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:24.345 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:24.342+0000 7f37f1c63640 10 monclient: shutdown 2024-03-25T18:34:24.346 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:24.342+0000 7f37f1c63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:24.346 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:24.342+0000 7f37f1c63640 1 -- 172.21.15.155:0/2717495002 >> v1:172.21.15.155:6790/0 conn(0x7f37ec13f710 legacy=0x7f37ec13bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:24.346 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:24.342+0000 7f37f1c63640 1 -- 172.21.15.155:0/2717495002 shutdown_connections 2024-03-25T18:34:24.346 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:24.342+0000 7f37f1c63640 1 -- 172.21.15.155:0/2717495002 >> 172.21.15.155:0/2717495002 conn(0x7f37ec0b7ea0 msgr2=0x7f37ec074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:24.346 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:24.343+0000 7f37f1c63640 1 -- 172.21.15.155:0/2717495002 shutdown_connections 2024-03-25T18:34:24.346 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:24.343+0000 7f37f1c63640 1 -- 172.21.15.155:0/2717495002 wait complete. 2024-03-25T18:34:24.346 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:24.343+0000 7f37f1c63640 1 librados: shutdown 2024-03-25T18:34:24.356 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:34:31.163 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:34:31.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.261+0000 7fd65277b640 10 monclient: get_monmap_and_config 2024-03-25T18:34:31.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.261+0000 7fd65277b640 10 monclient: build_initial_monmap 2024-03-25T18:34:31.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.261+0000 7fd65277b640 10 monclient: monmap: 2024-03-25T18:34:31.263 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:34:31.263 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:34:31.263 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:34:31.262756+0000 2024-03-25T18:34:31.263 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:34:31.262756+0000 2024-03-25T18:34:31.263 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:31.263 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:34:31.263 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:34:31.263 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:34:31.263 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:34:31.263 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:34:31.265 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.262+0000 7fd65277b640 1 Processor -- start 2024-03-25T18:34:31.265 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.262+0000 7fd65277b640 1 -- start start 2024-03-25T18:34:31.265 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.262+0000 7fd65277b640 10 monclient: init 2024-03-25T18:34:31.265 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.263+0000 7fd65277b640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:31.265 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.263+0000 7fd65277b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:34:31.265 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.263+0000 7fd65277b640 10 monclient(hunting): picked mon.noname-c con 0x7fd64c0a98f0 addr v1:172.21.15.167:6789/0 2024-03-25T18:34:31.265 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.263+0000 7fd65277b640 10 monclient(hunting): picked mon.noname-a con 0x7fd64c13f710 addr v1:172.21.15.155:6789/0 2024-03-25T18:34:31.265 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.263+0000 7fd65277b640 10 monclient(hunting): picked mon.noname-b con 0x7fd64c1430c0 addr v1:172.21.15.155:6790/0 2024-03-25T18:34:31.265 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.263+0000 7fd65277b640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd64c142520 con 0x7fd64c13f710 2024-03-25T18:34:31.265 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.263+0000 7fd65277b640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd64c0b28a0 con 0x7fd64c0a98f0 2024-03-25T18:34:31.265 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.263+0000 7fd65277b640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd64c0b2a30 con 0x7fd64c1430c0 2024-03-25T18:34:31.266 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.263+0000 7fd65277b640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:31.266 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.263+0000 7fd65277b640 10 monclient(hunting): authenticate will time out at 8117.573263s 2024-03-25T18:34:31.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.273+0000 7fd64b7fe640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fd64c13f710 0x7fd64c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:38136/0 (socket says 172.21.15.155:38136) 2024-03-25T18:34:31.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.273+0000 7fd64b7fe640 1 -- 172.21.15.155:0/1378809164 learned_addr learned my addr 172.21.15.155:0/1378809164 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:34:31.280 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.279+0000 7fd64affd640 1 -- 172.21.15.155:0/1378809164 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3688770815 0 0) 0x7fd64c142520 con 0x7fd64c13f710 2024-03-25T18:34:31.280 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.279+0000 7fd64affd640 10 monclient(hunting): _negotiate 2024-03-25T18:34:31.280 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.279+0000 7fd64affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:31.280 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.279+0000 7fd64affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:31.280 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.279+0000 7fd64affd640 10 monclient(hunting): my global_id is 4665 2024-03-25T18:34:31.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.279+0000 7fd64affd640 1 -- 172.21.15.155:0/1378809164 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd634003520 con 0x7fd64c13f710 2024-03-25T18:34:31.296 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.295+0000 7fd64affd640 1 -- 172.21.15.155:0/1378809164 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 217417396 0 0) 0x7fd634003520 con 0x7fd64c13f710 2024-03-25T18:34:31.296 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.295+0000 7fd64affd640 1 -- 172.21.15.155:0/1378809164 >> v1:172.21.15.155:6790/0 conn(0x7fd64c1430c0 legacy=0x7fd64c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:34:31.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.295+0000 7fd64affd640 1 -- 172.21.15.155:0/1378809164 >> v1:172.21.15.167:6789/0 conn(0x7fd64c0a98f0 legacy=0x7fd64c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:34:31.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.295+0000 7fd64affd640 10 monclient: _finish_hunting 0 2024-03-25T18:34:31.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.295+0000 7fd64affd640 1 monclient: found mon.noname-a 2024-03-25T18:34:31.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.295+0000 7fd64affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:31.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.295+0000 7fd64affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-25T18:34:31.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.295+0000 7fd64affd640 1 -- 172.21.15.155:0/1378809164 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd64c0b44c0 con 0x7fd64c13f710 2024-03-25T18:34:31.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.295+0000 7fd64affd640 10 monclient: _finish_auth 0 2024-03-25T18:34:31.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.295+0000 7fd64affd640 10 monclient: _check_auth_tickets 2024-03-25T18:34:31.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.295+0000 7fd64affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:31.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.295+0000 7fd64affd640 1 -- 172.21.15.155:0/1378809164 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fd640001690 con 0x7fd64c13f710 2024-03-25T18:34:31.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.295+0000 7fd64affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:31.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.295+0000 7fd64affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:34:31.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.295+0000 7fd64affd640 10 monclient: dump: 2024-03-25T18:34:31.298 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:34:31.298 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:34:31.298 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:31.298 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:31.298 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:31.298 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:34:31.298 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:34:31.298 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:34:31.298 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:34:31.298 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:34:31.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.295+0000 7fd65277b640 5 monclient: authenticate success, global_id 4665 2024-03-25T18:34:31.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.295+0000 7fd65277b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:31.311 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.310+0000 7fd64affd640 1 -- 172.21.15.155:0/1378809164 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd640001860 con 0x7fd64c13f710 2024-03-25T18:34:31.311 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.310+0000 7fd64affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:31.311 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.310+0000 7fd64affd640 1 -- 172.21.15.155:0/1378809164 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fd640001cb0 con 0x7fd64c13f710 2024-03-25T18:34:31.311 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.310+0000 7fd64affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:31.311 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.310+0000 7fd64affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:34:31.312 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.310+0000 7fd64affd640 10 monclient: dump: 2024-03-25T18:34:31.312 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:34:31.312 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:34:31.312 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:31.312 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:31.312 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:31.312 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:34:31.312 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:34:31.312 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:34:31.312 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:34:31.312 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:34:31.312 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.310+0000 7fd65277b640 10 monclient: get_monmap_and_config success 2024-03-25T18:34:31.312 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.310+0000 7fd65277b640 4 set_mon_vals no callback set 2024-03-25T18:34:31.314 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.311+0000 7fd65277b640 10 monclient: shutdown 2024-03-25T18:34:31.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.311+0000 7fd65277b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:31.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.311+0000 7fd65277b640 1 -- 172.21.15.155:0/1378809164 >> v1:172.21.15.155:6789/0 conn(0x7fd64c13f710 legacy=0x7fd64c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:31.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.311+0000 7fd65277b640 1 -- 172.21.15.155:0/1378809164 shutdown_connections 2024-03-25T18:34:31.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.311+0000 7fd65277b640 1 -- 172.21.15.155:0/1378809164 >> 172.21.15.155:0/1378809164 conn(0x7fd64c0b7ea0 msgr2=0x7fd64c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:31.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.311+0000 7fd65277b640 1 -- 172.21.15.155:0/1378809164 shutdown_connections 2024-03-25T18:34:31.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.311+0000 7fd65277b640 1 -- 172.21.15.155:0/1378809164 wait complete. 2024-03-25T18:34:31.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.311+0000 7fd65277b640 10 monclient: build_initial_monmap 2024-03-25T18:34:31.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.312+0000 7fd65277b640 10 monclient: monmap: 2024-03-25T18:34:31.315 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:34:31.315 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:34:31.315 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:34:31.316 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:34:31.316 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:31.316 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:34:31.316 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:34:31.316 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:34:31.316 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:34:31.316 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:34:31.316 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.312+0000 7fd65277b640 1 librados: starting msgr at 2024-03-25T18:34:31.316 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.312+0000 7fd65277b640 1 librados: starting objecter 2024-03-25T18:34:31.316 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.312+0000 7fd65277b640 1 Processor -- start 2024-03-25T18:34:31.316 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.312+0000 7fd65277b640 1 -- start start 2024-03-25T18:34:31.316 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.312+0000 7fd65277b640 1 librados: setting wanted keys 2024-03-25T18:34:31.316 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.312+0000 7fd65277b640 1 librados: calling monclient init 2024-03-25T18:34:31.317 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.312+0000 7fd65277b640 10 monclient: init 2024-03-25T18:34:31.317 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.313+0000 7fd65277b640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:31.317 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.313+0000 7fd65277b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:34:31.317 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.313+0000 7fd65277b640 10 monclient(hunting): picked mon.noname-b con 0x7fd64c0a98f0 addr v1:172.21.15.167:6789/0 2024-03-25T18:34:31.317 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.313+0000 7fd65277b640 10 monclient(hunting): picked mon.noname-c con 0x7fd64c13f710 addr v1:172.21.15.155:6790/0 2024-03-25T18:34:31.317 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.313+0000 7fd65277b640 10 monclient(hunting): picked mon.noname-a con 0x7fd64c1430c0 addr v1:172.21.15.155:6789/0 2024-03-25T18:34:31.317 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.313+0000 7fd65277b640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd64c071440 con 0x7fd64c1430c0 2024-03-25T18:34:31.317 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.313+0000 7fd65277b640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd64c071f00 con 0x7fd64c0a98f0 2024-03-25T18:34:31.317 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.313+0000 7fd65277b640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd64c072090 con 0x7fd64c13f710 2024-03-25T18:34:31.317 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.313+0000 7fd65277b640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:31.317 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.313+0000 7fd65277b640 10 monclient(hunting): authenticate will time out at 8117.623437s 2024-03-25T18:34:31.325 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.324+0000 7fd64bfff640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7fd64c0a98f0 0x7fd64c070b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:59710/0 (socket says 172.21.15.155:59710) 2024-03-25T18:34:31.325 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.324+0000 7fd64bfff640 1 -- 172.21.15.155:0/591856335 learned_addr learned my addr 172.21.15.155:0/591856335 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:34:31.327 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.326+0000 7fd648ff9640 10 client.?.objecter ms_handle_connect 0x7fd64c0a98f0 2024-03-25T18:34:31.327 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.326+0000 7fd648ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:31.332 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.331+0000 7fd648ff9640 10 client.?.objecter ms_handle_connect 0x7fd64c1430c0 2024-03-25T18:34:31.332 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.331+0000 7fd648ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:31.332 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.331+0000 7fd648ff9640 10 client.?.objecter ms_handle_connect 0x7fd64c13f710 2024-03-25T18:34:31.332 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.331+0000 7fd648ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:31.337 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.335+0000 7fd648ff9640 1 -- 172.21.15.155:0/591856335 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 138923809 0 0) 0x7fd64c071440 con 0x7fd64c1430c0 2024-03-25T18:34:31.337 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.335+0000 7fd648ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:34:31.337 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.335+0000 7fd648ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:31.337 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.335+0000 7fd648ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:31.337 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.335+0000 7fd648ff9640 10 monclient(hunting): my global_id is 4668 2024-03-25T18:34:31.337 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.335+0000 7fd648ff9640 1 -- 172.21.15.155:0/591856335 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd638004eb0 con 0x7fd64c1430c0 2024-03-25T18:34:31.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.340+0000 7fd648ff9640 1 -- 172.21.15.155:0/591856335 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1291175518 0 0) 0x7fd64c071f00 con 0x7fd64c0a98f0 2024-03-25T18:34:31.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.340+0000 7fd648ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:34:31.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.340+0000 7fd648ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:31.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.340+0000 7fd648ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:31.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.340+0000 7fd648ff9640 10 monclient(hunting): my global_id is 4663 2024-03-25T18:34:31.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.340+0000 7fd648ff9640 1 -- 172.21.15.155:0/591856335 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd64c071440 con 0x7fd64c0a98f0 2024-03-25T18:34:31.348 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.346+0000 7fd648ff9640 1 -- 172.21.15.155:0/591856335 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3181073235 0 0) 0x7fd638004eb0 con 0x7fd64c1430c0 2024-03-25T18:34:31.348 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.346+0000 7fd648ff9640 1 -- 172.21.15.155:0/591856335 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd64c071f00 con 0x7fd64c1430c0 2024-03-25T18:34:31.348 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.346+0000 7fd648ff9640 1 -- 172.21.15.155:0/591856335 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fd644001ad0 con 0x7fd64c1430c0 2024-03-25T18:34:31.348 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.346+0000 7fd648ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:31.348 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.346+0000 7fd648ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:34:31.348 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.346+0000 7fd648ff9640 10 monclient(hunting): dump: 2024-03-25T18:34:31.348 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:34:31.348 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:34:31.349 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:31.349 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:31.349 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:31.349 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:34:31.349 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:34:31.349 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:34:31.349 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:34:31.349 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:34:31.349 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.346+0000 7fd648ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:34:31.350 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.348+0000 7fd648ff9640 1 -- 172.21.15.155:0/591856335 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2862474379 0 0) 0x7fd64c072090 con 0x7fd64c13f710 2024-03-25T18:34:31.350 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.348+0000 7fd648ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:34:31.350 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.348+0000 7fd648ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:31.350 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.348+0000 7fd648ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:31.350 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.348+0000 7fd648ff9640 10 monclient(hunting): my global_id is 4676 2024-03-25T18:34:31.350 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.348+0000 7fd648ff9640 1 -- 172.21.15.155:0/591856335 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd638004eb0 con 0x7fd64c13f710 2024-03-25T18:34:31.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.358+0000 7fd648ff9640 1 -- 172.21.15.155:0/591856335 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1541819363 0 0) 0x7fd64c071f00 con 0x7fd64c1430c0 2024-03-25T18:34:31.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.359+0000 7fd648ff9640 1 -- 172.21.15.155:0/591856335 >> v1:172.21.15.155:6790/0 conn(0x7fd64c13f710 legacy=0x7fd64c138f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:31.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.359+0000 7fd648ff9640 1 -- 172.21.15.155:0/591856335 >> v1:172.21.15.167:6789/0 conn(0x7fd64c0a98f0 legacy=0x7fd64c070b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:31.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.359+0000 7fd648ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:34:31.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.359+0000 7fd648ff9640 1 monclient: found mon.a 2024-03-25T18:34:31.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.359+0000 7fd648ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:31.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.359+0000 7fd648ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:34:31.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.359+0000 7fd648ff9640 1 -- 172.21.15.155:0/591856335 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd64c06f0c0 con 0x7fd64c1430c0 2024-03-25T18:34:31.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.359+0000 7fd648ff9640 10 monclient: _finish_auth 0 2024-03-25T18:34:31.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.359+0000 7fd648ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:34:31.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.359+0000 7fd648ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:31.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.359+0000 7fd65277b640 5 monclient: authenticate success, global_id 4668 2024-03-25T18:34:31.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.359+0000 7fd65277b640 10 monclient: _renew_subs 2024-03-25T18:34:31.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.359+0000 7fd65277b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:34:31.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.359+0000 7fd65277b640 1 -- 172.21.15.155:0/591856335 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd64c06f390 con 0x7fd64c1430c0 2024-03-25T18:34:31.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.359+0000 7fd65277b640 10 client.4668.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:34:31.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.359+0000 7fd65277b640 10 monclient: _renew_subs 2024-03-25T18:34:31.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.359+0000 7fd65277b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:34:31.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.359+0000 7fd65277b640 1 -- 172.21.15.155:0/591856335 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd64c139850 con 0x7fd64c1430c0 2024-03-25T18:34:31.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.359+0000 7fd65277b640 1 librados: init done 2024-03-25T18:34:31.362 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.360+0000 7fd65277b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:31.362 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.360+0000 7fd65277b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:31.362 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.360+0000 7fd65277b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:34:31.362 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.360+0000 7fd65277b640 1 -- 172.21.15.155:0/591856335 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd64c071280 con 0x7fd64c1430c0 2024-03-25T18:34:31.375 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.373+0000 7fd648ff9640 1 -- 172.21.15.155:0/591856335 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd644001fe0 con 0x7fd64c1430c0 2024-03-25T18:34:31.375 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.373+0000 7fd648ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:31.375 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.373+0000 7fd648ff9640 1 -- 172.21.15.155:0/591856335 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fd644003b10 con 0x7fd64c1430c0 2024-03-25T18:34:31.375 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.373+0000 7fd648ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:31.375 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.373+0000 7fd648ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:34:31.375 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.373+0000 7fd648ff9640 10 monclient: dump: 2024-03-25T18:34:31.375 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:34:31.375 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:34:31.375 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:31.375 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:31.376 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:31.376 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:34:31.376 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:34:31.376 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:34:31.376 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:34:31.376 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:34:31.376 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.373+0000 7fd64a7fc640 4 set_mon_vals no callback set 2024-03-25T18:34:31.688 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.687+0000 7fd648ff9640 1 -- 172.21.15.155:0/591856335 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7fd644019ae0 con 0x7fd64c1430c0 2024-03-25T18:34:31.689 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.687+0000 7fd64bfff640 10 monclient: get_auth_request con 0x7fd63806dd60 auth_method 0 2024-03-25T18:34:31.689 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.688+0000 7fd648ff9640 10 client.4668.objecter ms_handle_connect 0x7fd63806dd60 2024-03-25T18:34:31.694 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.693+0000 7fd648ff9640 1 -- 172.21.15.155:0/591856335 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7fd644053850 con 0x7fd64c1430c0 2024-03-25T18:34:31.694 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.693+0000 7fd648ff9640 10 client.4668.objecter ms_dispatch 0x7fd64c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:34:31.694 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.693+0000 7fd648ff9640 3 client.4668.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:34:31.695 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.693+0000 7fd648ff9640 3 client.4668.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:34:31.695 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.693+0000 7fd648ff9640 20 client.4668.objecter dump_active .. 0 homeless 2024-03-25T18:34:31.695 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.693+0000 7fd648ff9640 1 -- 172.21.15.155:0/591856335 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd64404f190 con 0x7fd64c1430c0 2024-03-25T18:34:31.695 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.693+0000 7fd648ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:31.695 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.693+0000 7fd648ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:34:31.950 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.948+0000 7fd65277b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:31.950 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.948+0000 7fd65277b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:31.950 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.948+0000 7fd65277b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:34:31.950 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.948+0000 7fd65277b640 1 -- 172.21.15.155:0/591856335 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd64c0701e0 con 0x7fd64c1430c0 2024-03-25T18:34:31.977 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.976+0000 7fd648ff9640 1 -- 172.21.15.155:0/591856335 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7fd644051290 con 0x7fd64c1430c0 2024-03-25T18:34:31.977 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.976+0000 7fd648ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:31.977 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.976+0000 7fd648ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:34:31.977 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:34:31.978 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:34:31.980 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.978+0000 7fd65277b640 10 librados: watch_flush enter 2024-03-25T18:34:31.980 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.978+0000 7fd65277b640 10 librados: watch_flush exit 2024-03-25T18:34:31.981 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.978+0000 7fd65277b640 20 client.4668.objecter shutdown clearing up homeless session... 2024-03-25T18:34:31.981 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.978+0000 7fd65277b640 10 client.4668.objecter successfully canceled tick 2024-03-25T18:34:31.981 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.978+0000 7fd65277b640 1 -- 172.21.15.155:0/591856335 >> v1:172.21.15.167:6800/2919070503 conn(0x7fd63806dd60 legacy=0x7fd638070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:31.981 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.978+0000 7fd65277b640 10 monclient: shutdown 2024-03-25T18:34:31.981 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.978+0000 7fd65277b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:31.981 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.978+0000 7fd65277b640 1 -- 172.21.15.155:0/591856335 >> v1:172.21.15.155:6789/0 conn(0x7fd64c1430c0 legacy=0x7fd64c1decd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:31.981 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.978+0000 7fd65277b640 1 -- 172.21.15.155:0/591856335 shutdown_connections 2024-03-25T18:34:31.981 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.978+0000 7fd65277b640 1 -- 172.21.15.155:0/591856335 >> 172.21.15.155:0/591856335 conn(0x7fd64c0b7ea0 msgr2=0x7fd64c13ca80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:31.981 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.979+0000 7fd65277b640 1 -- 172.21.15.155:0/591856335 shutdown_connections 2024-03-25T18:34:31.981 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.979+0000 7fd65277b640 1 -- 172.21.15.155:0/591856335 wait complete. 2024-03-25T18:34:31.981 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:31.979+0000 7fd65277b640 1 librados: shutdown 2024-03-25T18:34:31.993 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:34:32.464 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:10:06.068 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:32.476 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:10:06.080 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:32.506 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:10:06.110 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:32.520 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:10:06.099 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:32.524 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:10:06.103 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:32.528 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:10:06.133 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:32.532 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:10:06.137 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:32.548 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:10:06.127 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:32.567 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:10:06.172 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:38.800 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:34:38.919 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.918+0000 7f085a8c1640 10 monclient: get_monmap_and_config 2024-03-25T18:34:38.920 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.918+0000 7f085a8c1640 10 monclient: build_initial_monmap 2024-03-25T18:34:38.920 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.918+0000 7f085a8c1640 10 monclient: monmap: 2024-03-25T18:34:38.920 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:34:38.920 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:34:38.920 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:34:38.919709+0000 2024-03-25T18:34:38.920 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:34:38.919709+0000 2024-03-25T18:34:38.920 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:38.920 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:34:38.920 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:34:38.920 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:34:38.920 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:34:38.920 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:34:38.922 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.919+0000 7f085a8c1640 1 Processor -- start 2024-03-25T18:34:38.922 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.919+0000 7f085a8c1640 1 -- start start 2024-03-25T18:34:38.922 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.919+0000 7f085a8c1640 10 monclient: init 2024-03-25T18:34:38.922 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.920+0000 7f085a8c1640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:38.922 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.920+0000 7f085a8c1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:34:38.922 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.920+0000 7f085a8c1640 10 monclient(hunting): picked mon.noname-a con 0x7f08540a98f0 addr v1:172.21.15.155:6789/0 2024-03-25T18:34:38.922 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.920+0000 7f085a8c1640 10 monclient(hunting): picked mon.noname-b con 0x7f085413f710 addr v1:172.21.15.155:6790/0 2024-03-25T18:34:38.922 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.920+0000 7f085a8c1640 10 monclient(hunting): picked mon.noname-c con 0x7f08541430c0 addr v1:172.21.15.167:6789/0 2024-03-25T18:34:38.922 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.920+0000 7f085a8c1640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0854142520 con 0x7f08540a98f0 2024-03-25T18:34:38.922 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.920+0000 7f085a8c1640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08540b28a0 con 0x7f08541430c0 2024-03-25T18:34:38.923 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.920+0000 7f085a8c1640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08540b2a30 con 0x7f085413f710 2024-03-25T18:34:38.923 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.920+0000 7f085a8c1640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:38.923 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.920+0000 7f085a8c1640 10 monclient(hunting): authenticate will time out at 8125.230223s 2024-03-25T18:34:38.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.930+0000 7f0858e5e640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f08541430c0 0x7f085414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:34924/0 (socket says 172.21.15.155:34924) 2024-03-25T18:34:38.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.930+0000 7f0858e5e640 1 -- 172.21.15.155:0/4007407013 learned_addr learned my addr 172.21.15.155:0/4007407013 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:34:38.952 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.951+0000 7f0852ffd640 1 -- 172.21.15.155:0/4007407013 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4205220200 0 0) 0x7f08540b28a0 con 0x7f08541430c0 2024-03-25T18:34:38.952 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.951+0000 7f0852ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:34:38.952 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.951+0000 7f0852ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:38.952 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.951+0000 7f0852ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:38.952 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.951+0000 7f0852ffd640 10 monclient(hunting): my global_id is 4666 2024-03-25T18:34:38.953 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.951+0000 7f0852ffd640 1 -- 172.21.15.155:0/4007407013 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f083c003520 con 0x7f08541430c0 2024-03-25T18:34:38.961 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.960+0000 7f0852ffd640 1 -- 172.21.15.155:0/4007407013 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 858002598 0 0) 0x7f08540b2a30 con 0x7f085413f710 2024-03-25T18:34:38.961 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.960+0000 7f0852ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:34:38.962 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.960+0000 7f0852ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:38.962 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.960+0000 7f0852ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:38.962 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.960+0000 7f0852ffd640 10 monclient(hunting): my global_id is 4679 2024-03-25T18:34:38.962 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.960+0000 7f0852ffd640 1 -- 172.21.15.155:0/4007407013 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08540b28a0 con 0x7f085413f710 2024-03-25T18:34:38.968 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.967+0000 7f0852ffd640 1 -- 172.21.15.155:0/4007407013 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1516747622 0 0) 0x7f0854142520 con 0x7f08540a98f0 2024-03-25T18:34:38.968 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.967+0000 7f0852ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:34:38.968 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.967+0000 7f0852ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:38.968 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.967+0000 7f0852ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:38.968 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.967+0000 7f0852ffd640 10 monclient(hunting): my global_id is 4671 2024-03-25T18:34:38.968 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.967+0000 7f0852ffd640 1 -- 172.21.15.155:0/4007407013 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08540b2a30 con 0x7f08540a98f0 2024-03-25T18:34:38.971 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.969+0000 7f0852ffd640 1 -- 172.21.15.155:0/4007407013 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 981002719 0 0) 0x7f083c003520 con 0x7f08541430c0 2024-03-25T18:34:38.971 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.969+0000 7f0852ffd640 1 -- 172.21.15.155:0/4007407013 >> v1:172.21.15.155:6790/0 conn(0x7f085413f710 legacy=0x7f08541429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:38.971 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.969+0000 7f0852ffd640 1 -- 172.21.15.155:0/4007407013 >> v1:172.21.15.155:6789/0 conn(0x7f08540a98f0 legacy=0x7f08540a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:38.971 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.969+0000 7f0852ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:34:38.971 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.969+0000 7f0852ffd640 1 monclient: found mon.noname-c 2024-03-25T18:34:38.971 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.969+0000 7f0852ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:38.971 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.969+0000 7f0852ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.167:6789/0 2024-03-25T18:34:38.971 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.969+0000 7f0852ffd640 1 -- 172.21.15.155:0/4007407013 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08540b44c0 con 0x7f08541430c0 2024-03-25T18:34:38.971 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.969+0000 7f0852ffd640 10 monclient: _finish_auth 0 2024-03-25T18:34:38.971 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.969+0000 7f0852ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:34:38.972 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.969+0000 7f0852ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:38.972 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.969+0000 7f0852ffd640 1 -- 172.21.15.155:0/4007407013 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f084c001a10 con 0x7f08541430c0 2024-03-25T18:34:38.972 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.969+0000 7f0852ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:38.972 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.969+0000 7f0852ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:34:38.972 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.969+0000 7f0852ffd640 10 monclient: dump: 2024-03-25T18:34:38.972 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:34:38.972 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:34:38.972 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:38.972 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:38.972 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:38.972 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:34:38.972 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:34:38.972 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:34:38.972 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:34:38.972 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:34:38.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.970+0000 7f085a8c1640 5 monclient: authenticate success, global_id 4666 2024-03-25T18:34:38.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.970+0000 7f085a8c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:38.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.989+0000 7f0852ffd640 1 -- 172.21.15.155:0/4007407013 <== mon.1 v1:172.21.15.167:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f084c001b70 con 0x7f08541430c0 2024-03-25T18:34:38.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.989+0000 7f0852ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:38.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.989+0000 7f0852ffd640 1 -- 172.21.15.155:0/4007407013 <== mon.1 v1:172.21.15.167:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f084c001f80 con 0x7f08541430c0 2024-03-25T18:34:38.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.989+0000 7f0852ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:38.991 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.989+0000 7f0852ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:38.991 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.989+0000 7f0852ffd640 10 monclient: dump: 2024-03-25T18:34:38.991 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:34:38.991 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:34:38.991 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:38.991 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:38.991 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:38.991 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:34:38.991 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:34:38.991 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:34:38.991 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:34:38.991 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:34:38.991 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.989+0000 7f085a8c1640 10 monclient: get_monmap_and_config success 2024-03-25T18:34:38.991 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.989+0000 7f085a8c1640 4 set_mon_vals no callback set 2024-03-25T18:34:38.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.990+0000 7f085a8c1640 10 monclient: shutdown 2024-03-25T18:34:38.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.990+0000 7f085a8c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:38.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.990+0000 7f085a8c1640 1 -- 172.21.15.155:0/4007407013 >> v1:172.21.15.167:6789/0 conn(0x7f08541430c0 legacy=0x7f085414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:38.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.990+0000 7f085a8c1640 1 -- 172.21.15.155:0/4007407013 shutdown_connections 2024-03-25T18:34:38.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.990+0000 7f085a8c1640 1 -- 172.21.15.155:0/4007407013 >> 172.21.15.155:0/4007407013 conn(0x7f08540b7ea0 msgr2=0x7f08540b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:38.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.990+0000 7f085a8c1640 1 -- 172.21.15.155:0/4007407013 shutdown_connections 2024-03-25T18:34:38.995 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.990+0000 7f085a8c1640 1 -- 172.21.15.155:0/4007407013 wait complete. 2024-03-25T18:34:38.995 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.991+0000 7f085a8c1640 10 monclient: build_initial_monmap 2024-03-25T18:34:38.995 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.991+0000 7f085a8c1640 10 monclient: monmap: 2024-03-25T18:34:38.995 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:34:38.995 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:34:38.995 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:34:38.995 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:34:38.995 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:38.995 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:34:38.995 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:34:38.995 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:34:38.995 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:34:38.995 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:34:38.995 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.991+0000 7f085a8c1640 1 librados: starting msgr at 2024-03-25T18:34:38.995 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.991+0000 7f085a8c1640 1 librados: starting objecter 2024-03-25T18:34:38.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.991+0000 7f085a8c1640 1 Processor -- start 2024-03-25T18:34:38.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.991+0000 7f085a8c1640 1 -- start start 2024-03-25T18:34:38.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.991+0000 7f085a8c1640 1 librados: setting wanted keys 2024-03-25T18:34:38.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.991+0000 7f085a8c1640 1 librados: calling monclient init 2024-03-25T18:34:38.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.991+0000 7f085a8c1640 10 monclient: init 2024-03-25T18:34:38.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.992+0000 7f085a8c1640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:38.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.992+0000 7f085a8c1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:34:38.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.992+0000 7f085a8c1640 10 monclient(hunting): picked mon.noname-b con 0x7f08540a98f0 addr v1:172.21.15.167:6789/0 2024-03-25T18:34:38.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.992+0000 7f085a8c1640 10 monclient(hunting): picked mon.noname-c con 0x7f085413f710 addr v1:172.21.15.155:6790/0 2024-03-25T18:34:38.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.992+0000 7f085a8c1640 10 monclient(hunting): picked mon.noname-a con 0x7f08541430c0 addr v1:172.21.15.155:6789/0 2024-03-25T18:34:38.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.992+0000 7f085a8c1640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08541daca0 con 0x7f08541430c0 2024-03-25T18:34:38.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.992+0000 7f085a8c1640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0854138ab0 con 0x7f08540a98f0 2024-03-25T18:34:38.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.992+0000 7f085a8c1640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08540aba90 con 0x7f085413f710 2024-03-25T18:34:38.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.992+0000 7f085a8c1640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:38.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.992+0000 7f085a8c1640 10 monclient(hunting): authenticate will time out at 8125.302406s 2024-03-25T18:34:38.997 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.997+0000 7f0853fff640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f08540a98f0 0x7f08541383a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:34968/0 (socket says 172.21.15.155:34968) 2024-03-25T18:34:38.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.997+0000 7f0853fff640 1 -- 172.21.15.155:0/433962263 learned_addr learned my addr 172.21.15.155:0/433962263 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:34:38.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.997+0000 7f0850ff9640 10 client.?.objecter ms_handle_connect 0x7f08540a98f0 2024-03-25T18:34:38.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:38.997+0000 7f0850ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:39.017 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.016+0000 7f0850ff9640 10 client.?.objecter ms_handle_connect 0x7f08541430c0 2024-03-25T18:34:39.017 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.016+0000 7f0850ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:39.023 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.022+0000 7f0850ff9640 1 -- 172.21.15.155:0/433962263 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4018726406 0 0) 0x7f0854138ab0 con 0x7f08540a98f0 2024-03-25T18:34:39.023 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.022+0000 7f0850ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:34:39.023 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.022+0000 7f0850ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:39.023 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.022+0000 7f0850ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:39.023 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.022+0000 7f0850ff9640 10 monclient(hunting): my global_id is 4669 2024-03-25T18:34:39.023 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.022+0000 7f0850ff9640 1 -- 172.21.15.155:0/433962263 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0840004eb0 con 0x7f08540a98f0 2024-03-25T18:34:39.023 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.022+0000 7f0850ff9640 1 -- 172.21.15.155:0/433962263 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3647674924 0 0) 0x7f08541daca0 con 0x7f08541430c0 2024-03-25T18:34:39.023 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.022+0000 7f0850ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:34:39.024 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.022+0000 7f0850ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:39.024 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.022+0000 7f0850ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:39.024 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.022+0000 7f0850ff9640 10 monclient(hunting): my global_id is 4674 2024-03-25T18:34:39.024 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.022+0000 7f0850ff9640 1 -- 172.21.15.155:0/433962263 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0854138ab0 con 0x7f08541430c0 2024-03-25T18:34:39.024 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.023+0000 7f0850ff9640 10 client.?.objecter ms_handle_connect 0x7f085413f710 2024-03-25T18:34:39.024 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.023+0000 7f0850ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:39.035 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.034+0000 7f0850ff9640 1 -- 172.21.15.155:0/433962263 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 432146889 0 0) 0x7f08540aba90 con 0x7f085413f710 2024-03-25T18:34:39.035 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.034+0000 7f0850ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:34:39.035 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.034+0000 7f0850ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:39.035 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.034+0000 7f0850ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:39.035 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.034+0000 7f0850ff9640 10 monclient(hunting): my global_id is 4682 2024-03-25T18:34:39.035 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.034+0000 7f0850ff9640 1 -- 172.21.15.155:0/433962263 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08541daca0 con 0x7f085413f710 2024-03-25T18:34:39.045 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.044+0000 7f0850ff9640 1 -- 172.21.15.155:0/433962263 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 101499385 0 0) 0x7f0840004eb0 con 0x7f08540a98f0 2024-03-25T18:34:39.046 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.044+0000 7f0850ff9640 1 -- 172.21.15.155:0/433962263 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f08540aba90 con 0x7f08540a98f0 2024-03-25T18:34:39.046 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.044+0000 7f0850ff9640 1 -- 172.21.15.155:0/433962263 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f0844001a10 con 0x7f08540a98f0 2024-03-25T18:34:39.046 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.044+0000 7f0850ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:39.046 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.044+0000 7f0850ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:34:39.046 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.044+0000 7f0850ff9640 10 monclient(hunting): dump: 2024-03-25T18:34:39.046 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:34:39.046 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:34:39.046 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:39.046 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:39.046 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:39.046 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:34:39.046 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:34:39.047 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:34:39.047 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:34:39.047 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:34:39.047 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.045+0000 7f0850ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:34:39.053 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.052+0000 7f0850ff9640 1 -- 172.21.15.155:0/433962263 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2655928323 0 0) 0x7f0854138ab0 con 0x7f08541430c0 2024-03-25T18:34:39.053 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.052+0000 7f0850ff9640 1 -- 172.21.15.155:0/433962263 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0840004eb0 con 0x7f08541430c0 2024-03-25T18:34:39.053 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.052+0000 7f0850ff9640 1 -- 172.21.15.155:0/433962263 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f084c001e80 con 0x7f08541430c0 2024-03-25T18:34:39.053 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.052+0000 7f0850ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:39.053 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.052+0000 7f0850ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:34:39.053 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.052+0000 7f0850ff9640 10 monclient(hunting): dump: 2024-03-25T18:34:39.053 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:34:39.053 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:34:39.053 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:39.053 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:39.053 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:39.053 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:34:39.053 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:34:39.054 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:34:39.054 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:34:39.054 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:34:39.062 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.061+0000 7f0850ff9640 1 -- 172.21.15.155:0/433962263 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3456024080 0 0) 0x7f08541daca0 con 0x7f085413f710 2024-03-25T18:34:39.062 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.061+0000 7f0850ff9640 1 -- 172.21.15.155:0/433962263 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0854138ab0 con 0x7f085413f710 2024-03-25T18:34:39.062 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.061+0000 7f0850ff9640 1 -- 172.21.15.155:0/433962263 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f0848001930 con 0x7f085413f710 2024-03-25T18:34:39.062 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.061+0000 7f0850ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:39.062 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.061+0000 7f0850ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:34:39.062 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.061+0000 7f0850ff9640 10 monclient(hunting): dump: 2024-03-25T18:34:39.062 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:34:39.062 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:34:39.062 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:39.063 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:39.063 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:39.063 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:34:39.063 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:34:39.063 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:34:39.063 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:34:39.063 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:34:39.074 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.072+0000 7f0850ff9640 1 -- 172.21.15.155:0/433962263 <== mon.1 v1:172.21.15.167:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2264218981 0 0) 0x7f08540aba90 con 0x7f08540a98f0 2024-03-25T18:34:39.074 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.072+0000 7f0850ff9640 1 -- 172.21.15.155:0/433962263 >> v1:172.21.15.155:6790/0 conn(0x7f085413f710 legacy=0x7f08541db130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:39.074 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.072+0000 7f0850ff9640 1 -- 172.21.15.155:0/433962263 >> v1:172.21.15.155:6789/0 conn(0x7f08541430c0 legacy=0x7f08541d47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:39.074 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.072+0000 7f0850ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:34:39.074 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.072+0000 7f0850ff9640 1 monclient: found mon.b 2024-03-25T18:34:39.074 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.072+0000 7f0850ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:39.074 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.072+0000 7f0850ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:34:39.074 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.072+0000 7f0850ff9640 1 -- 172.21.15.155:0/433962263 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08541d41e0 con 0x7f08540a98f0 2024-03-25T18:34:39.074 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.072+0000 7f0850ff9640 10 monclient: _finish_auth 0 2024-03-25T18:34:39.074 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.072+0000 7f0850ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:34:39.074 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.072+0000 7f0850ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:39.074 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.072+0000 7f085a8c1640 5 monclient: authenticate success, global_id 4669 2024-03-25T18:34:39.075 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.072+0000 7f085a8c1640 10 monclient: _renew_subs 2024-03-25T18:34:39.075 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.072+0000 7f085a8c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:34:39.075 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.072+0000 7f085a8c1640 1 -- 172.21.15.155:0/433962263 --> v1:172.21.15.167:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08541db840 con 0x7f08540a98f0 2024-03-25T18:34:39.075 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.072+0000 7f085a8c1640 10 client.4669.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:34:39.075 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.072+0000 7f085a8c1640 10 monclient: _renew_subs 2024-03-25T18:34:39.075 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.072+0000 7f085a8c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:34:39.075 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.072+0000 7f085a8c1640 1 -- 172.21.15.155:0/433962263 --> v1:172.21.15.167:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0854138f00 con 0x7f08540a98f0 2024-03-25T18:34:39.075 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.072+0000 7f085a8c1640 1 librados: init done 2024-03-25T18:34:39.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.073+0000 7f085a8c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:39.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.073+0000 7f085a8c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:39.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.073+0000 7f085a8c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:34:39.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.073+0000 7f085a8c1640 1 -- 172.21.15.155:0/433962263 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08540627b0 con 0x7f08540a98f0 2024-03-25T18:34:39.385 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.384+0000 7f0850ff9640 1 -- 172.21.15.155:0/433962263 <== mon.1 v1:172.21.15.167:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0844001c70 con 0x7f08540a98f0 2024-03-25T18:34:39.386 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.384+0000 7f0850ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:39.386 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.384+0000 7f0850ff9640 1 -- 172.21.15.155:0/433962263 <== mon.1 v1:172.21.15.167:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f0844003a60 con 0x7f08540a98f0 2024-03-25T18:34:39.386 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.384+0000 7f0850ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:39.386 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.384+0000 7f0850ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:39.386 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.384+0000 7f0850ff9640 10 monclient: dump: 2024-03-25T18:34:39.386 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:34:39.386 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:34:39.386 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:39.386 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:39.386 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:39.386 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:34:39.386 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:34:39.387 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:34:39.387 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:34:39.387 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:34:39.387 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.384+0000 7f08527fc640 4 set_mon_vals no callback set 2024-03-25T18:34:39.387 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.385+0000 7f0850ff9640 1 -- 172.21.15.155:0/433962263 <== mon.1 v1:172.21.15.167:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7f0844003c80 con 0x7f08540a98f0 2024-03-25T18:34:39.387 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.385+0000 7f08537fe640 10 monclient: get_auth_request con 0x7f084006e110 auth_method 0 2024-03-25T18:34:39.387 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.385+0000 7f0850ff9640 10 client.4669.objecter ms_handle_connect 0x7f084006e110 2024-03-25T18:34:39.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.389+0000 7f0850ff9640 1 -- 172.21.15.155:0/433962263 <== mon.1 v1:172.21.15.167:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7f08440548b0 con 0x7f08540a98f0 2024-03-25T18:34:39.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.389+0000 7f0850ff9640 10 client.4669.objecter ms_dispatch 0x7f0854000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:34:39.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.389+0000 7f0850ff9640 3 client.4669.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:34:39.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.389+0000 7f0850ff9640 3 client.4669.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:34:39.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.389+0000 7f0850ff9640 20 client.4669.objecter dump_active .. 0 homeless 2024-03-25T18:34:39.391 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.390+0000 7f0850ff9640 1 -- 172.21.15.155:0/433962263 <== mon.1 v1:172.21.15.167:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f08440503c0 con 0x7f08540a98f0 2024-03-25T18:34:39.391 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.390+0000 7f0850ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:39.391 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.390+0000 7f0850ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:34:39.626 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.624+0000 7f085a8c1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:39.626 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.624+0000 7f085a8c1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:39.626 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.624+0000 7f085a8c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:34:39.626 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.624+0000 7f085a8c1640 1 -- 172.21.15.155:0/433962263 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f08541d4370 con 0x7f08540a98f0 2024-03-25T18:34:39.658 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.657+0000 7f0850ff9640 1 -- 172.21.15.155:0/433962263 <== mon.1 v1:172.21.15.167:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7f08440522f0 con 0x7f08540a98f0 2024-03-25T18:34:39.658 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.657+0000 7f0850ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:39.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.657+0000 7f0850ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:34:39.659 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:34:39.659 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:34:39.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.659+0000 7f085a8c1640 10 librados: watch_flush enter 2024-03-25T18:34:39.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.659+0000 7f085a8c1640 10 librados: watch_flush exit 2024-03-25T18:34:39.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.659+0000 7f085a8c1640 20 client.4669.objecter shutdown clearing up homeless session... 2024-03-25T18:34:39.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.659+0000 7f085a8c1640 10 client.4669.objecter successfully canceled tick 2024-03-25T18:34:39.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.660+0000 7f085a8c1640 1 -- 172.21.15.155:0/433962263 >> v1:172.21.15.167:6800/2919070503 conn(0x7f084006e110 legacy=0x7f08400705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:39.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.660+0000 7f085a8c1640 10 monclient: shutdown 2024-03-25T18:34:39.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.660+0000 7f085a8c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:39.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.660+0000 7f085a8c1640 1 -- 172.21.15.155:0/433962263 >> v1:172.21.15.167:6789/0 conn(0x7f08540a98f0 legacy=0x7f08541383a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:39.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.660+0000 7f085a8c1640 1 -- 172.21.15.155:0/433962263 shutdown_connections 2024-03-25T18:34:39.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.660+0000 7f085a8c1640 1 -- 172.21.15.155:0/433962263 >> 172.21.15.155:0/433962263 conn(0x7f08540b7ea0 msgr2=0x7f0854074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:39.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.660+0000 7f085a8c1640 1 -- 172.21.15.155:0/433962263 shutdown_connections 2024-03-25T18:34:39.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.660+0000 7f085a8c1640 1 -- 172.21.15.155:0/433962263 wait complete. 2024-03-25T18:34:39.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:39.660+0000 7f085a8c1640 1 librados: shutdown 2024-03-25T18:34:39.673 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:34:46.479 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:34:46.596 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.595+0000 7fca7daf1640 10 monclient: get_monmap_and_config 2024-03-25T18:34:46.596 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.595+0000 7fca7daf1640 10 monclient: build_initial_monmap 2024-03-25T18:34:46.596 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.595+0000 7fca7daf1640 10 monclient: monmap: 2024-03-25T18:34:46.596 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:34:46.596 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:34:46.596 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:34:46.596213+0000 2024-03-25T18:34:46.596 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:34:46.596213+0000 2024-03-25T18:34:46.596 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:46.596 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:34:46.597 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:34:46.597 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:34:46.597 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:34:46.597 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:34:46.597 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.596+0000 7fca7daf1640 1 Processor -- start 2024-03-25T18:34:46.597 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.596+0000 7fca7daf1640 1 -- start start 2024-03-25T18:34:46.597 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.596+0000 7fca7daf1640 10 monclient: init 2024-03-25T18:34:46.598 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.597+0000 7fca7daf1640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:46.599 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.597+0000 7fca7daf1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:34:46.599 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.597+0000 7fca7daf1640 10 monclient(hunting): picked mon.noname-a con 0x7fca780a98f0 addr v1:172.21.15.155:6789/0 2024-03-25T18:34:46.599 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.597+0000 7fca7daf1640 10 monclient(hunting): picked mon.noname-b con 0x7fca7813f710 addr v1:172.21.15.155:6790/0 2024-03-25T18:34:46.599 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.597+0000 7fca7daf1640 10 monclient(hunting): picked mon.noname-c con 0x7fca781430c0 addr v1:172.21.15.167:6789/0 2024-03-25T18:34:46.599 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.597+0000 7fca7daf1640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca78142520 con 0x7fca780a98f0 2024-03-25T18:34:46.599 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.597+0000 7fca7daf1640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca780b28a0 con 0x7fca781430c0 2024-03-25T18:34:46.599 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.597+0000 7fca7daf1640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca780b2a30 con 0x7fca7813f710 2024-03-25T18:34:46.599 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.597+0000 7fca7daf1640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:46.599 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.597+0000 7fca7daf1640 10 monclient(hunting): authenticate will time out at 8132.907296s 2024-03-25T18:34:46.608 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.607+0000 7fca76ffd640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fca7813f710 0x7fca781429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:38758/0 (socket says 172.21.15.155:38758) 2024-03-25T18:34:46.608 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.607+0000 7fca76ffd640 1 -- 172.21.15.155:0/3236076798 learned_addr learned my addr 172.21.15.155:0/3236076798 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:34:46.619 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.618+0000 7fca767fc640 1 -- 172.21.15.155:0/3236076798 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3365008994 0 0) 0x7fca780b2a30 con 0x7fca7813f710 2024-03-25T18:34:46.619 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.618+0000 7fca767fc640 10 monclient(hunting): _negotiate 2024-03-25T18:34:46.619 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.618+0000 7fca767fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:46.619 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.618+0000 7fca767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:46.619 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.618+0000 7fca767fc640 10 monclient(hunting): my global_id is 4685 2024-03-25T18:34:46.619 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.618+0000 7fca767fc640 1 -- 172.21.15.155:0/3236076798 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca60003520 con 0x7fca7813f710 2024-03-25T18:34:46.625 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.624+0000 7fca767fc640 1 -- 172.21.15.155:0/3236076798 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1276732839 0 0) 0x7fca780b28a0 con 0x7fca781430c0 2024-03-25T18:34:46.625 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.624+0000 7fca767fc640 10 monclient(hunting): _negotiate 2024-03-25T18:34:46.625 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.624+0000 7fca767fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:46.625 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.624+0000 7fca767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:46.625 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.624+0000 7fca767fc640 10 monclient(hunting): my global_id is 4672 2024-03-25T18:34:46.625 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.624+0000 7fca767fc640 1 -- 172.21.15.155:0/3236076798 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca780b2a30 con 0x7fca781430c0 2024-03-25T18:34:46.630 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.628+0000 7fca767fc640 1 -- 172.21.15.155:0/3236076798 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2005025094 0 0) 0x7fca60003520 con 0x7fca7813f710 2024-03-25T18:34:46.630 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.628+0000 7fca767fc640 1 -- 172.21.15.155:0/3236076798 >> v1:172.21.15.167:6789/0 conn(0x7fca781430c0 legacy=0x7fca7814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:46.630 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.628+0000 7fca767fc640 1 -- 172.21.15.155:0/3236076798 >> v1:172.21.15.155:6789/0 conn(0x7fca780a98f0 legacy=0x7fca780a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:46.630 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.628+0000 7fca767fc640 10 monclient: _finish_hunting 0 2024-03-25T18:34:46.630 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.628+0000 7fca767fc640 1 monclient: found mon.noname-b 2024-03-25T18:34:46.630 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.628+0000 7fca767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:46.630 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.628+0000 7fca767fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-25T18:34:46.631 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.628+0000 7fca767fc640 1 -- 172.21.15.155:0/3236076798 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca780b44c0 con 0x7fca7813f710 2024-03-25T18:34:46.631 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.628+0000 7fca767fc640 10 monclient: _finish_auth 0 2024-03-25T18:34:46.631 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.628+0000 7fca767fc640 10 monclient: _check_auth_tickets 2024-03-25T18:34:46.631 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.628+0000 7fca767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:46.631 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.628+0000 7fca767fc640 1 -- 172.21.15.155:0/3236076798 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fca6c001690 con 0x7fca7813f710 2024-03-25T18:34:46.631 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.628+0000 7fca767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:46.631 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.628+0000 7fca767fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:34:46.631 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.628+0000 7fca767fc640 10 monclient: dump: 2024-03-25T18:34:46.631 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:34:46.631 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:34:46.631 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:46.631 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:46.631 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:46.631 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:34:46.631 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:34:46.632 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:34:46.632 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:34:46.632 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:34:46.632 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.628+0000 7fca7daf1640 5 monclient: authenticate success, global_id 4685 2024-03-25T18:34:46.632 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.629+0000 7fca7daf1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:46.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.646+0000 7fca767fc640 1 -- 172.21.15.155:0/3236076798 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca6c0017f0 con 0x7fca7813f710 2024-03-25T18:34:46.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.646+0000 7fca767fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:46.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.646+0000 7fca767fc640 1 -- 172.21.15.155:0/3236076798 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fca6c001c00 con 0x7fca7813f710 2024-03-25T18:34:46.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.646+0000 7fca7daf1640 10 monclient: get_monmap_and_config success 2024-03-25T18:34:46.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.646+0000 7fca7daf1640 4 set_mon_vals no callback set 2024-03-25T18:34:46.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.647+0000 7fca767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:46.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.647+0000 7fca767fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:34:46.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.647+0000 7fca767fc640 10 monclient: dump: 2024-03-25T18:34:46.649 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:34:46.649 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:34:46.649 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:46.649 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:46.649 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:46.649 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:34:46.649 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:34:46.649 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:34:46.649 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:34:46.649 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:34:46.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.647+0000 7fca7daf1640 10 monclient: shutdown 2024-03-25T18:34:46.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.647+0000 7fca7daf1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:46.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.647+0000 7fca7daf1640 1 -- 172.21.15.155:0/3236076798 >> v1:172.21.15.155:6790/0 conn(0x7fca7813f710 legacy=0x7fca781429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:46.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.647+0000 7fca7daf1640 1 -- 172.21.15.155:0/3236076798 shutdown_connections 2024-03-25T18:34:46.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.648+0000 7fca7daf1640 1 -- 172.21.15.155:0/3236076798 >> 172.21.15.155:0/3236076798 conn(0x7fca780b7ea0 msgr2=0x7fca780b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:46.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.648+0000 7fca7daf1640 1 -- 172.21.15.155:0/3236076798 shutdown_connections 2024-03-25T18:34:46.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.648+0000 7fca7daf1640 1 -- 172.21.15.155:0/3236076798 wait complete. 2024-03-25T18:34:46.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.648+0000 7fca7daf1640 10 monclient: build_initial_monmap 2024-03-25T18:34:46.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.648+0000 7fca7daf1640 10 monclient: monmap: 2024-03-25T18:34:46.651 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:34:46.651 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:34:46.651 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:34:46.651 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:34:46.651 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:46.651 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:34:46.651 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:34:46.651 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:34:46.651 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:34:46.651 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:34:46.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.649+0000 7fca7daf1640 1 librados: starting msgr at 2024-03-25T18:34:46.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.649+0000 7fca7daf1640 1 librados: starting objecter 2024-03-25T18:34:46.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.649+0000 7fca7daf1640 1 Processor -- start 2024-03-25T18:34:46.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.649+0000 7fca7daf1640 1 -- start start 2024-03-25T18:34:46.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.649+0000 7fca7daf1640 1 librados: setting wanted keys 2024-03-25T18:34:46.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.649+0000 7fca7daf1640 1 librados: calling monclient init 2024-03-25T18:34:46.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.649+0000 7fca7daf1640 10 monclient: init 2024-03-25T18:34:46.653 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.650+0000 7fca7daf1640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:46.653 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.650+0000 7fca7daf1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:34:46.653 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.650+0000 7fca7daf1640 10 monclient(hunting): picked mon.noname-a con 0x7fca780a98f0 addr v1:172.21.15.155:6789/0 2024-03-25T18:34:46.653 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.650+0000 7fca7daf1640 10 monclient(hunting): picked mon.noname-c con 0x7fca7813f710 addr v1:172.21.15.155:6790/0 2024-03-25T18:34:46.653 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.650+0000 7fca7daf1640 10 monclient(hunting): picked mon.noname-b con 0x7fca781430c0 addr v1:172.21.15.167:6789/0 2024-03-25T18:34:46.653 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.650+0000 7fca7daf1640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca78137df0 con 0x7fca780a98f0 2024-03-25T18:34:46.653 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.650+0000 7fca7daf1640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca781337a0 con 0x7fca781430c0 2024-03-25T18:34:46.653 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.650+0000 7fca7daf1640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca78133930 con 0x7fca7813f710 2024-03-25T18:34:46.653 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.650+0000 7fca7daf1640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:46.653 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.650+0000 7fca7daf1640 10 monclient(hunting): authenticate will time out at 8132.960356s 2024-03-25T18:34:46.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.663+0000 7fca77fff640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7fca781430c0 0x7fca78133090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:54172/0 (socket says 172.21.15.155:54172) 2024-03-25T18:34:46.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.663+0000 7fca77fff640 1 -- 172.21.15.155:0/1241761309 learned_addr learned my addr 172.21.15.155:0/1241761309 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:34:46.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.665+0000 7fca5ffff640 10 client.?.objecter ms_handle_connect 0x7fca781430c0 2024-03-25T18:34:46.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.665+0000 7fca5ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:46.670 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.669+0000 7fca5ffff640 10 client.?.objecter ms_handle_connect 0x7fca780a98f0 2024-03-25T18:34:46.670 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.669+0000 7fca5ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:46.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.670+0000 7fca5ffff640 10 client.?.objecter ms_handle_connect 0x7fca7813f710 2024-03-25T18:34:46.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.670+0000 7fca5ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:46.680 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.679+0000 7fca5ffff640 1 -- 172.21.15.155:0/1241761309 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2032947935 0 0) 0x7fca78133930 con 0x7fca7813f710 2024-03-25T18:34:46.680 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.679+0000 7fca5ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:34:46.681 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.679+0000 7fca5ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:46.681 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.679+0000 7fca5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:46.681 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.679+0000 7fca5ffff640 10 monclient(hunting): my global_id is 4688 2024-03-25T18:34:46.681 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.679+0000 7fca5ffff640 1 -- 172.21.15.155:0/1241761309 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca64004eb0 con 0x7fca7813f710 2024-03-25T18:34:46.681 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.680+0000 7fca5ffff640 1 -- 172.21.15.155:0/1241761309 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1698657153 0 0) 0x7fca781337a0 con 0x7fca781430c0 2024-03-25T18:34:46.681 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.680+0000 7fca5ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:34:46.681 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.680+0000 7fca5ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:46.681 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.680+0000 7fca5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:46.682 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.680+0000 7fca5ffff640 10 monclient(hunting): my global_id is 4675 2024-03-25T18:34:46.682 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.680+0000 7fca5ffff640 1 -- 172.21.15.155:0/1241761309 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca78133930 con 0x7fca781430c0 2024-03-25T18:34:46.683 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.682+0000 7fca5ffff640 1 -- 172.21.15.155:0/1241761309 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 106655844 0 0) 0x7fca78137df0 con 0x7fca780a98f0 2024-03-25T18:34:46.683 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.682+0000 7fca5ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:34:46.683 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.682+0000 7fca5ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:46.683 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.682+0000 7fca5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:46.684 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.682+0000 7fca5ffff640 10 monclient(hunting): my global_id is 4680 2024-03-25T18:34:46.684 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.682+0000 7fca5ffff640 1 -- 172.21.15.155:0/1241761309 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca781337a0 con 0x7fca780a98f0 2024-03-25T18:34:46.702 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.701+0000 7fca5ffff640 1 -- 172.21.15.155:0/1241761309 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2099235652 0 0) 0x7fca78133930 con 0x7fca781430c0 2024-03-25T18:34:46.702 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.701+0000 7fca5ffff640 1 -- 172.21.15.155:0/1241761309 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca78137df0 con 0x7fca781430c0 2024-03-25T18:34:46.702 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.701+0000 7fca5ffff640 1 -- 172.21.15.155:0/1241761309 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fca70001c10 con 0x7fca781430c0 2024-03-25T18:34:46.702 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.701+0000 7fca5ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:46.702 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.701+0000 7fca5ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:34:46.702 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.701+0000 7fca5ffff640 10 monclient(hunting): dump: 2024-03-25T18:34:46.702 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:34:46.702 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:34:46.702 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:46.702 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:46.703 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:46.703 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:34:46.703 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:34:46.703 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:34:46.703 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:34:46.703 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:34:46.703 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.701+0000 7fca5ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:34:46.711 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.710+0000 7fca5ffff640 1 -- 172.21.15.155:0/1241761309 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4264293996 0 0) 0x7fca64004eb0 con 0x7fca7813f710 2024-03-25T18:34:46.711 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.710+0000 7fca5ffff640 1 -- 172.21.15.155:0/1241761309 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca78133930 con 0x7fca7813f710 2024-03-25T18:34:46.711 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.710+0000 7fca5ffff640 1 -- 172.21.15.155:0/1241761309 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fca6c001bc0 con 0x7fca7813f710 2024-03-25T18:34:46.711 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.710+0000 7fca5ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:46.711 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.710+0000 7fca5ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:34:46.711 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.710+0000 7fca5ffff640 10 monclient(hunting): dump: 2024-03-25T18:34:46.712 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:34:46.712 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:34:46.712 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:46.712 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:46.712 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:46.712 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:34:46.712 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:34:46.712 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:34:46.712 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:34:46.712 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:34:46.718 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.716+0000 7fca5ffff640 1 -- 172.21.15.155:0/1241761309 <== mon.1 v1:172.21.15.167:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3668532005 0 0) 0x7fca78137df0 con 0x7fca781430c0 2024-03-25T18:34:46.718 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.716+0000 7fca5ffff640 1 -- 172.21.15.155:0/1241761309 >> v1:172.21.15.155:6790/0 conn(0x7fca7813f710 legacy=0x7fca781376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:46.718 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.716+0000 7fca5ffff640 1 -- 172.21.15.155:0/1241761309 >> v1:172.21.15.155:6789/0 conn(0x7fca780a98f0 legacy=0x7fca781392c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:46.718 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.716+0000 7fca5ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:34:46.718 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.716+0000 7fca5ffff640 1 monclient: found mon.b 2024-03-25T18:34:46.718 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.716+0000 7fca5ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:46.718 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.716+0000 7fca5ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:34:46.718 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.716+0000 7fca5ffff640 1 -- 172.21.15.155:0/1241761309 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca781367d0 con 0x7fca781430c0 2024-03-25T18:34:46.718 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.716+0000 7fca5ffff640 10 monclient: _finish_auth 0 2024-03-25T18:34:46.718 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.716+0000 7fca5ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:34:46.718 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.716+0000 7fca5ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:46.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.716+0000 7fca7daf1640 5 monclient: authenticate success, global_id 4675 2024-03-25T18:34:46.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.716+0000 7fca7daf1640 10 monclient: _renew_subs 2024-03-25T18:34:46.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.716+0000 7fca7daf1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:34:46.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.716+0000 7fca7daf1640 1 -- 172.21.15.155:0/1241761309 --> v1:172.21.15.167:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca781369e0 con 0x7fca781430c0 2024-03-25T18:34:46.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.716+0000 7fca7daf1640 10 client.4675.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:34:46.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.716+0000 7fca7daf1640 10 monclient: _renew_subs 2024-03-25T18:34:46.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.716+0000 7fca7daf1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:34:46.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.716+0000 7fca7daf1640 1 -- 172.21.15.155:0/1241761309 --> v1:172.21.15.167:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fca7813b640 con 0x7fca781430c0 2024-03-25T18:34:46.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.716+0000 7fca7daf1640 1 librados: init done 2024-03-25T18:34:46.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.717+0000 7fca7daf1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:46.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.717+0000 7fca7daf1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:46.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.717+0000 7fca7daf1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:34:46.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:46.717+0000 7fca7daf1640 1 -- 172.21.15.155:0/1241761309 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca780627b0 con 0x7fca781430c0 2024-03-25T18:34:47.027 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:47.026+0000 7fca5ffff640 1 -- 172.21.15.155:0/1241761309 <== mon.1 v1:172.21.15.167:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca70003990 con 0x7fca781430c0 2024-03-25T18:34:47.027 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:47.026+0000 7fca5ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:47.027 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:47.026+0000 7fca5ffff640 1 -- 172.21.15.155:0/1241761309 <== mon.1 v1:172.21.15.167:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fca70002100 con 0x7fca781430c0 2024-03-25T18:34:47.027 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:47.026+0000 7fca5ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:47.027 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:47.026+0000 7fca5ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:47.027 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:47.026+0000 7fca5ffff640 10 monclient: dump: 2024-03-25T18:34:47.027 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:34:47.027 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:34:47.028 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:47.028 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:47.028 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:47.028 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:34:47.028 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:34:47.028 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:34:47.028 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:34:47.028 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:34:47.028 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:47.026+0000 7fca75ffb640 4 set_mon_vals no callback set 2024-03-25T18:34:47.029 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:47.027+0000 7fca5ffff640 1 -- 172.21.15.155:0/1241761309 <== mon.1 v1:172.21.15.167:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7fca70003de0 con 0x7fca781430c0 2024-03-25T18:34:47.029 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:47.027+0000 7fca777fe640 10 monclient: get_auth_request con 0x7fca6406df70 auth_method 0 2024-03-25T18:34:47.029 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:47.028+0000 7fca5ffff640 10 client.4675.objecter ms_handle_connect 0x7fca6406df70 2024-03-25T18:34:47.033 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:47.032+0000 7fca5ffff640 1 -- 172.21.15.155:0/1241761309 <== mon.1 v1:172.21.15.167:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4068722799 0 0) 0x7fca700047e0 con 0x7fca781430c0 2024-03-25T18:34:47.033 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:47.032+0000 7fca5ffff640 10 client.4675.objecter ms_dispatch 0x7fca78000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:34:47.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:47.032+0000 7fca5ffff640 3 client.4675.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:34:47.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:47.032+0000 7fca5ffff640 3 client.4675.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:34:47.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:47.032+0000 7fca5ffff640 20 client.4675.objecter dump_active .. 0 homeless 2024-03-25T18:34:47.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:47.033+0000 7fca5ffff640 1 -- 172.21.15.155:0/1241761309 <== mon.1 v1:172.21.15.167:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fca70050560 con 0x7fca781430c0 2024-03-25T18:34:47.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:47.033+0000 7fca5ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:47.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:47.033+0000 7fca5ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:34:47.269 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:47.268+0000 7fca7daf1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:47.270 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:47.268+0000 7fca7daf1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:47.270 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:47.268+0000 7fca7daf1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:34:47.270 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:47.268+0000 7fca7daf1640 1 -- 172.21.15.155:0/1241761309 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fca78074cc0 con 0x7fca781430c0 2024-03-25T18:34:47.319 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:47.318+0000 7fca5ffff640 1 -- 172.21.15.155:0/1241761309 <== mon.1 v1:172.21.15.167:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4237933640) 0x7fca70052490 con 0x7fca781430c0 2024-03-25T18:34:47.319 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:47.318+0000 7fca5ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:47.319 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:47.318+0000 7fca5ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:34:47.319 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:34:47.319 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:24:48.507564+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:48.507564+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:34:47.322 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:47.320+0000 7fca7daf1640 10 librados: watch_flush enter 2024-03-25T18:34:47.322 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:47.320+0000 7fca7daf1640 10 librados: watch_flush exit 2024-03-25T18:34:47.322 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:47.320+0000 7fca7daf1640 20 client.4675.objecter shutdown clearing up homeless session... 2024-03-25T18:34:47.322 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:47.320+0000 7fca7daf1640 10 client.4675.objecter successfully canceled tick 2024-03-25T18:34:47.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:47.321+0000 7fca7daf1640 1 -- 172.21.15.155:0/1241761309 >> v1:172.21.15.167:6800/2919070503 conn(0x7fca6406df70 legacy=0x7fca64070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:47.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:47.321+0000 7fca7daf1640 10 monclient: shutdown 2024-03-25T18:34:47.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:47.321+0000 7fca7daf1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:47.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:47.321+0000 7fca7daf1640 1 -- 172.21.15.155:0/1241761309 >> v1:172.21.15.167:6789/0 conn(0x7fca781430c0 legacy=0x7fca78133090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:47.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:47.321+0000 7fca7daf1640 1 -- 172.21.15.155:0/1241761309 shutdown_connections 2024-03-25T18:34:47.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:47.321+0000 7fca7daf1640 1 -- 172.21.15.155:0/1241761309 >> 172.21.15.155:0/1241761309 conn(0x7fca780b7ea0 msgr2=0x7fca78074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:47.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:47.321+0000 7fca7daf1640 1 -- 172.21.15.155:0/1241761309 shutdown_connections 2024-03-25T18:34:47.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:47.321+0000 7fca7daf1640 1 -- 172.21.15.155:0/1241761309 wait complete. 2024-03-25T18:34:47.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:47.321+0000 7fca7daf1640 1 librados: shutdown 2024-03-25T18:34:47.335 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:34:52.550 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:10:26.154 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:52.560 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:10:26.164 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:52.585 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:10:26.190 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:52.601 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:10:26.180 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:52.609 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:10:26.188 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:52.610 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:10:26.215 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:52.614 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:10:26.219 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:52.643 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:10:26.221 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:52.656 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:10:26.261 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:54.140 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:34:54.256 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.255+0000 7f111fce6640 10 monclient: get_monmap_and_config 2024-03-25T18:34:54.256 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.255+0000 7f111fce6640 10 monclient: build_initial_monmap 2024-03-25T18:34:54.256 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.255+0000 7f111fce6640 10 monclient: monmap: 2024-03-25T18:34:54.256 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:34:54.257 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:34:54.257 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:34:54.256552+0000 2024-03-25T18:34:54.257 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:34:54.256552+0000 2024-03-25T18:34:54.257 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:54.257 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:34:54.257 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:34:54.257 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:34:54.257 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:34:54.257 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:34:54.258 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.256+0000 7f111fce6640 1 Processor -- start 2024-03-25T18:34:54.258 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.256+0000 7f111fce6640 1 -- start start 2024-03-25T18:34:54.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.256+0000 7f111fce6640 10 monclient: init 2024-03-25T18:34:54.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.257+0000 7f111fce6640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:54.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.257+0000 7f111fce6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:34:54.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.257+0000 7f111fce6640 10 monclient(hunting): picked mon.noname-a con 0x7f11180a98f0 addr v1:172.21.15.155:6789/0 2024-03-25T18:34:54.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.257+0000 7f111fce6640 10 monclient(hunting): picked mon.noname-c con 0x7f111813f710 addr v1:172.21.15.167:6789/0 2024-03-25T18:34:54.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.257+0000 7f111fce6640 10 monclient(hunting): picked mon.noname-b con 0x7f11181430c0 addr v1:172.21.15.155:6790/0 2024-03-25T18:34:54.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.257+0000 7f111fce6640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1118142520 con 0x7f11180a98f0 2024-03-25T18:34:54.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.257+0000 7f111fce6640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11180b28a0 con 0x7f111813f710 2024-03-25T18:34:54.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.257+0000 7f111fce6640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11180b2a30 con 0x7f11181430c0 2024-03-25T18:34:54.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.257+0000 7f111fce6640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:54.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.257+0000 7f111fce6640 10 monclient(hunting): authenticate will time out at 8140.567068s 2024-03-25T18:34:54.265 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.264+0000 7f111d281640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f111813f710 0x7f11181429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:58782/0 (socket says 172.21.15.155:58782) 2024-03-25T18:34:54.265 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.264+0000 7f111d281640 1 -- 172.21.15.155:0/3175239694 learned_addr learned my addr 172.21.15.155:0/3175239694 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:34:54.277 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.276+0000 7f111ca80640 1 -- 172.21.15.155:0/3175239694 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1248359902 0 0) 0x7f11180b28a0 con 0x7f111813f710 2024-03-25T18:34:54.277 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.276+0000 7f111ca80640 10 monclient(hunting): _negotiate 2024-03-25T18:34:54.277 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.276+0000 7f111ca80640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:54.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.276+0000 7f111ca80640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:54.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.276+0000 7f111ca80640 10 monclient(hunting): my global_id is 4678 2024-03-25T18:34:54.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.276+0000 7f111ca80640 1 -- 172.21.15.155:0/3175239694 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1104003520 con 0x7f111813f710 2024-03-25T18:34:54.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.286+0000 7f111ca80640 1 -- 172.21.15.155:0/3175239694 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3557418561 0 0) 0x7f1104003520 con 0x7f111813f710 2024-03-25T18:34:54.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.286+0000 7f111ca80640 1 -- 172.21.15.155:0/3175239694 >> v1:172.21.15.155:6790/0 conn(0x7f11181430c0 legacy=0x7f111814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:54.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.286+0000 7f111ca80640 1 -- 172.21.15.155:0/3175239694 >> v1:172.21.15.155:6789/0 conn(0x7f11180a98f0 legacy=0x7f11180a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:54.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.286+0000 7f111ca80640 10 monclient: _finish_hunting 0 2024-03-25T18:34:54.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.286+0000 7f111ca80640 1 monclient: found mon.noname-c 2024-03-25T18:34:54.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.286+0000 7f111ca80640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:54.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.286+0000 7f111ca80640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.167:6789/0 2024-03-25T18:34:54.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.286+0000 7f111ca80640 1 -- 172.21.15.155:0/3175239694 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11180b44c0 con 0x7f111813f710 2024-03-25T18:34:54.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.286+0000 7f111ca80640 10 monclient: _finish_auth 0 2024-03-25T18:34:54.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.286+0000 7f111ca80640 10 monclient: _check_auth_tickets 2024-03-25T18:34:54.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.286+0000 7f111ca80640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:54.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.286+0000 7f111fce6640 5 monclient: authenticate success, global_id 4678 2024-03-25T18:34:54.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.286+0000 7f111fce6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:54.290 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.289+0000 7f111ca80640 1 -- 172.21.15.155:0/3175239694 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f1108001690 con 0x7f111813f710 2024-03-25T18:34:54.290 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.289+0000 7f111ca80640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:54.290 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.289+0000 7f111ca80640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:34:54.290 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.289+0000 7f111ca80640 10 monclient: dump: 2024-03-25T18:34:54.290 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:34:54.290 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:34:54.290 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:54.290 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:54.290 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:54.290 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:34:54.290 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:34:54.291 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:34:54.291 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:34:54.291 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:34:54.291 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.289+0000 7f111fce6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:54.293 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.292+0000 7f111ca80640 1 -- 172.21.15.155:0/3175239694 <== mon.1 v1:172.21.15.167:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1108001920 con 0x7f111813f710 2024-03-25T18:34:54.293 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.292+0000 7f111ca80640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:54.293 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.292+0000 7f111ca80640 1 -- 172.21.15.155:0/3175239694 <== mon.1 v1:172.21.15.167:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f1108001cb0 con 0x7f111813f710 2024-03-25T18:34:54.293 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.292+0000 7f111fce6640 10 monclient: get_monmap_and_config success 2024-03-25T18:34:54.293 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.292+0000 7f111fce6640 4 set_mon_vals no callback set 2024-03-25T18:34:54.294 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.292+0000 7f111ca80640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:54.294 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.292+0000 7f111ca80640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:54.294 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.292+0000 7f111ca80640 10 monclient: dump: 2024-03-25T18:34:54.294 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:34:54.294 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:34:54.294 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:54.294 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:54.294 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:54.294 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:34:54.294 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:34:54.294 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:34:54.294 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:34:54.294 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:34:54.296 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.293+0000 7f111fce6640 10 monclient: shutdown 2024-03-25T18:34:54.296 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.293+0000 7f111fce6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:54.296 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.293+0000 7f111fce6640 1 -- 172.21.15.155:0/3175239694 >> v1:172.21.15.167:6789/0 conn(0x7f111813f710 legacy=0x7f11181429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:54.296 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.293+0000 7f111fce6640 1 -- 172.21.15.155:0/3175239694 shutdown_connections 2024-03-25T18:34:54.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.293+0000 7f111fce6640 1 -- 172.21.15.155:0/3175239694 >> 172.21.15.155:0/3175239694 conn(0x7f11180b7ea0 msgr2=0x7f11180b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:54.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.293+0000 7f111fce6640 1 -- 172.21.15.155:0/3175239694 shutdown_connections 2024-03-25T18:34:54.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.293+0000 7f111fce6640 1 -- 172.21.15.155:0/3175239694 wait complete. 2024-03-25T18:34:54.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.294+0000 7f111fce6640 10 monclient: build_initial_monmap 2024-03-25T18:34:54.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.294+0000 7f111fce6640 10 monclient: monmap: 2024-03-25T18:34:54.297 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:34:54.297 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:34:54.297 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:34:54.297 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:34:54.297 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:54.297 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:34:54.297 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:34:54.297 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:34:54.297 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:34:54.297 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:34:54.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.294+0000 7f111fce6640 1 librados: starting msgr at 2024-03-25T18:34:54.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.294+0000 7f111fce6640 1 librados: starting objecter 2024-03-25T18:34:54.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.294+0000 7f111fce6640 1 Processor -- start 2024-03-25T18:34:54.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.294+0000 7f111fce6640 1 -- start start 2024-03-25T18:34:54.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.294+0000 7f111fce6640 1 librados: setting wanted keys 2024-03-25T18:34:54.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.294+0000 7f111fce6640 1 librados: calling monclient init 2024-03-25T18:34:54.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.294+0000 7f111fce6640 10 monclient: init 2024-03-25T18:34:54.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.295+0000 7f111fce6640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:54.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.295+0000 7f111fce6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:34:54.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.295+0000 7f111fce6640 10 monclient(hunting): picked mon.noname-c con 0x7f11180a98f0 addr v1:172.21.15.155:6790/0 2024-03-25T18:34:54.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.295+0000 7f111fce6640 10 monclient(hunting): picked mon.noname-b con 0x7f111813f710 addr v1:172.21.15.167:6789/0 2024-03-25T18:34:54.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.295+0000 7f111fce6640 10 monclient(hunting): picked mon.noname-a con 0x7f11181430c0 addr v1:172.21.15.155:6789/0 2024-03-25T18:34:54.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.295+0000 7f111fce6640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11181333c0 con 0x7f11181430c0 2024-03-25T18:34:54.299 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.295+0000 7f111fce6640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1118135100 con 0x7f111813f710 2024-03-25T18:34:54.299 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.295+0000 7f111fce6640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1118132f50 con 0x7f11180a98f0 2024-03-25T18:34:54.299 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.295+0000 7f111fce6640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:54.299 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.295+0000 7f111fce6640 10 monclient(hunting): authenticate will time out at 8140.605405s 2024-03-25T18:34:54.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.307+0000 7f111d281640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f111813f710 0x7f11181db110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:58820/0 (socket says 172.21.15.155:58820) 2024-03-25T18:34:54.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.307+0000 7f111d281640 1 -- 172.21.15.155:0/1497330581 learned_addr learned my addr 172.21.15.155:0/1497330581 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:34:54.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.307+0000 7f110e7fc640 10 client.?.objecter ms_handle_connect 0x7f111813f710 2024-03-25T18:34:54.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.307+0000 7f110e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:54.311 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.310+0000 7f110e7fc640 10 client.?.objecter ms_handle_connect 0x7f11180a98f0 2024-03-25T18:34:54.311 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.310+0000 7f110e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:54.317 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.316+0000 7f110e7fc640 10 client.?.objecter ms_handle_connect 0x7f11181430c0 2024-03-25T18:34:54.317 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.316+0000 7f110e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:54.317 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.316+0000 7f110e7fc640 1 -- 172.21.15.155:0/1497330581 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2180260087 0 0) 0x7f1118135100 con 0x7f111813f710 2024-03-25T18:34:54.317 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.316+0000 7f110e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:34:54.317 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.316+0000 7f110e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:54.317 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.316+0000 7f110e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:54.317 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.316+0000 7f110e7fc640 10 monclient(hunting): my global_id is 4681 2024-03-25T18:34:54.317 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.316+0000 7f110e7fc640 1 -- 172.21.15.155:0/1497330581 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10fc004eb0 con 0x7f111813f710 2024-03-25T18:34:54.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.317+0000 7f110e7fc640 1 -- 172.21.15.155:0/1497330581 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2531819461 0 0) 0x7f1118132f50 con 0x7f11180a98f0 2024-03-25T18:34:54.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.317+0000 7f110e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:34:54.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.317+0000 7f110e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:54.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.317+0000 7f110e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:54.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.317+0000 7f110e7fc640 10 monclient(hunting): my global_id is 4694 2024-03-25T18:34:54.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.317+0000 7f110e7fc640 1 -- 172.21.15.155:0/1497330581 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1118135100 con 0x7f11180a98f0 2024-03-25T18:34:54.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.329+0000 7f110e7fc640 1 -- 172.21.15.155:0/1497330581 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2948738024 0 0) 0x7f10fc004eb0 con 0x7f111813f710 2024-03-25T18:34:54.331 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.329+0000 7f110e7fc640 1 -- 172.21.15.155:0/1497330581 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1118132f50 con 0x7f111813f710 2024-03-25T18:34:54.331 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.329+0000 7f110e7fc640 1 -- 172.21.15.155:0/1497330581 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f1108001650 con 0x7f111813f710 2024-03-25T18:34:54.331 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.329+0000 7f110e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:54.331 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.329+0000 7f110e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:34:54.331 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.329+0000 7f110e7fc640 10 monclient(hunting): dump: 2024-03-25T18:34:54.331 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:34:54.331 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:34:54.331 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:54.331 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:54.331 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:54.331 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:34:54.331 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:34:54.331 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:34:54.331 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:34:54.331 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:34:54.332 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.329+0000 7f110e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:34:54.332 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.330+0000 7f110e7fc640 1 -- 172.21.15.155:0/1497330581 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3170187031 0 0) 0x7f11181333c0 con 0x7f11181430c0 2024-03-25T18:34:54.332 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.330+0000 7f110e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:34:54.332 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.330+0000 7f110e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:54.332 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.330+0000 7f110e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:54.332 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.330+0000 7f110e7fc640 10 monclient(hunting): my global_id is 4686 2024-03-25T18:34:54.332 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.330+0000 7f110e7fc640 1 -- 172.21.15.155:0/1497330581 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10fc004eb0 con 0x7f11181430c0 2024-03-25T18:34:54.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.341+0000 7f110e7fc640 1 -- 172.21.15.155:0/1497330581 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 258043191 0 0) 0x7f1118135100 con 0x7f11180a98f0 2024-03-25T18:34:54.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.341+0000 7f110e7fc640 1 -- 172.21.15.155:0/1497330581 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11181333c0 con 0x7f11180a98f0 2024-03-25T18:34:54.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.341+0000 7f110e7fc640 1 -- 172.21.15.155:0/1497330581 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f11140018b0 con 0x7f11180a98f0 2024-03-25T18:34:54.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.341+0000 7f110e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:54.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.341+0000 7f110e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:34:54.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.341+0000 7f110e7fc640 10 monclient(hunting): dump: 2024-03-25T18:34:54.342 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:34:54.342 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:34:54.343 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:54.343 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:54.343 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:54.343 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:34:54.343 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:34:54.343 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:34:54.343 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:34:54.343 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:34:54.345 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.343+0000 7f110e7fc640 1 -- 172.21.15.155:0/1497330581 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4185495096 0 0) 0x7f10fc004eb0 con 0x7f11181430c0 2024-03-25T18:34:54.346 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.343+0000 7f110e7fc640 1 -- 172.21.15.155:0/1497330581 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1118135100 con 0x7f11181430c0 2024-03-25T18:34:54.346 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.343+0000 7f110e7fc640 1 -- 172.21.15.155:0/1497330581 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f1110001a60 con 0x7f11181430c0 2024-03-25T18:34:54.346 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.343+0000 7f110e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:54.346 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.343+0000 7f110e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:34:54.346 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.343+0000 7f110e7fc640 10 monclient(hunting): dump: 2024-03-25T18:34:54.346 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:34:54.346 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:34:54.346 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:54.346 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:54.347 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:54.347 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:34:54.347 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:34:54.347 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:34:54.347 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:34:54.347 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:34:54.347 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.344+0000 7f110e7fc640 1 -- 172.21.15.155:0/1497330581 <== mon.1 v1:172.21.15.167:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3747460917 0 0) 0x7f1118132f50 con 0x7f111813f710 2024-03-25T18:34:54.347 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.344+0000 7f110e7fc640 1 -- 172.21.15.155:0/1497330581 >> v1:172.21.15.155:6790/0 conn(0x7f11180a98f0 legacy=0x7f11181daa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:54.347 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.344+0000 7f110e7fc640 1 -- 172.21.15.155:0/1497330581 >> v1:172.21.15.155:6789/0 conn(0x7f11181430c0 legacy=0x7f11181e1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:54.347 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.344+0000 7f110e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:34:54.347 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.344+0000 7f110e7fc640 1 monclient: found mon.b 2024-03-25T18:34:54.347 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.344+0000 7f110e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:54.348 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.344+0000 7f110e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:34:54.348 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.344+0000 7f110e7fc640 1 -- 172.21.15.155:0/1497330581 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11181330e0 con 0x7f111813f710 2024-03-25T18:34:54.348 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.344+0000 7f110e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:34:54.348 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.344+0000 7f110e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:34:54.348 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.344+0000 7f110e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:54.348 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.344+0000 7f111fce6640 5 monclient: authenticate success, global_id 4681 2024-03-25T18:34:54.348 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.344+0000 7f111fce6640 10 monclient: _renew_subs 2024-03-25T18:34:54.348 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.344+0000 7f111fce6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:34:54.348 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.344+0000 7f111fce6640 1 -- 172.21.15.155:0/1497330581 --> v1:172.21.15.167:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1118135420 con 0x7f111813f710 2024-03-25T18:34:54.348 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.344+0000 7f111fce6640 10 client.4681.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:34:54.349 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.344+0000 7f111fce6640 10 monclient: _renew_subs 2024-03-25T18:34:54.349 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.344+0000 7f111fce6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:34:54.349 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.344+0000 7f111fce6640 1 -- 172.21.15.155:0/1497330581 --> v1:172.21.15.167:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f111813bb10 con 0x7f111813f710 2024-03-25T18:34:54.349 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.344+0000 7f111fce6640 1 librados: init done 2024-03-25T18:34:54.349 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.345+0000 7f111fce6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:54.349 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.345+0000 7f111fce6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:54.350 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.345+0000 7f111fce6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:34:54.350 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.345+0000 7f111fce6640 1 -- 172.21.15.155:0/1497330581 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1118133570 con 0x7f111813f710 2024-03-25T18:34:54.620 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.619+0000 7f110e7fc640 1 -- 172.21.15.155:0/1497330581 <== mon.1 v1:172.21.15.167:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1108004bf0 con 0x7f111813f710 2024-03-25T18:34:54.621 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.619+0000 7f110e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:54.621 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.619+0000 7f110e7fc640 1 -- 172.21.15.155:0/1497330581 <== mon.1 v1:172.21.15.167:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f1108004d60 con 0x7f111813f710 2024-03-25T18:34:54.621 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.619+0000 7f110e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:54.621 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.619+0000 7f110e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:54.621 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.619+0000 7f110e7fc640 10 monclient: dump: 2024-03-25T18:34:54.621 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:34:54.621 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:34:54.621 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:54.621 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:34:54.621 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:54.622 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:34:54.622 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:34:54.622 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:34:54.622 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:34:54.622 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:34:54.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.619+0000 7f110ffff640 4 set_mon_vals no callback set 2024-03-25T18:34:54.623 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.620+0000 7f110e7fc640 1 -- 172.21.15.155:0/1497330581 <== mon.1 v1:172.21.15.167:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7f110801ad30 con 0x7f111813f710 2024-03-25T18:34:54.623 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.621+0000 7f111da82640 10 monclient: get_auth_request con 0x7f10fc06e0c0 auth_method 0 2024-03-25T18:34:54.623 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.621+0000 7f110e7fc640 10 client.4681.objecter ms_handle_connect 0x7f10fc06e0c0 2024-03-25T18:34:54.627 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.626+0000 7f110e7fc640 1 -- 172.21.15.155:0/1497330581 <== mon.1 v1:172.21.15.167:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1555029372 0 0) 0x7f1108054800 con 0x7f111813f710 2024-03-25T18:34:54.627 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.626+0000 7f110e7fc640 10 client.4681.objecter ms_dispatch 0x7f1118000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:34:54.627 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.626+0000 7f110e7fc640 3 client.4681.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:34:54.627 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.626+0000 7f110e7fc640 3 client.4681.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:34:54.627 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.626+0000 7f110e7fc640 20 client.4681.objecter dump_active .. 0 homeless 2024-03-25T18:34:54.627 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.626+0000 7f110e7fc640 1 -- 172.21.15.155:0/1497330581 <== mon.1 v1:172.21.15.167:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1108050310 con 0x7f111813f710 2024-03-25T18:34:54.628 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.626+0000 7f110e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:54.628 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.626+0000 7f110e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:34:54.869 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.868+0000 7f111fce6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:54.869 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.868+0000 7f111fce6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:54.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.868+0000 7f111fce6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:34:54.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.868+0000 7f111fce6640 1 -- 172.21.15.155:0/1497330581 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f11181357c0 con 0x7f111813f710 2024-03-25T18:34:54.889 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.888+0000 7f110e7fc640 1 -- 172.21.15.155:0/1497330581 <== mon.1 v1:172.21.15.167:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3170230695) 0x7f1108052240 con 0x7f111813f710 2024-03-25T18:34:54.889 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.888+0000 7f110e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:54.889 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.888+0000 7f110e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:34:54.889 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:34:54.889 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":11,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:34:47.433406+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:34:47.433406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:34:54.891 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.890+0000 7f111fce6640 10 librados: watch_flush enter 2024-03-25T18:34:54.891 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.890+0000 7f111fce6640 10 librados: watch_flush exit 2024-03-25T18:34:54.891 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.890+0000 7f111fce6640 20 client.4681.objecter shutdown clearing up homeless session... 2024-03-25T18:34:54.891 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.890+0000 7f111fce6640 10 client.4681.objecter successfully canceled tick 2024-03-25T18:34:54.891 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.890+0000 7f111fce6640 1 -- 172.21.15.155:0/1497330581 >> v1:172.21.15.167:6800/2919070503 conn(0x7f10fc06e0c0 legacy=0x7f10fc070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:54.891 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.890+0000 7f111fce6640 10 monclient: shutdown 2024-03-25T18:34:54.891 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.890+0000 7f111fce6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:54.891 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.890+0000 7f111fce6640 1 -- 172.21.15.155:0/1497330581 >> v1:172.21.15.167:6789/0 conn(0x7f111813f710 legacy=0x7f11181db110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:54.892 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.891+0000 7f111fce6640 1 -- 172.21.15.155:0/1497330581 shutdown_connections 2024-03-25T18:34:54.892 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.891+0000 7f111fce6640 1 -- 172.21.15.155:0/1497330581 >> 172.21.15.155:0/1497330581 conn(0x7f11180b7ea0 msgr2=0x7f1118074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:54.892 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.891+0000 7f111fce6640 1 -- 172.21.15.155:0/1497330581 shutdown_connections 2024-03-25T18:34:54.892 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.891+0000 7f111fce6640 1 -- 172.21.15.155:0/1497330581 wait complete. 2024-03-25T18:34:54.892 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:34:54.891+0000 7f111fce6640 1 librados: shutdown 2024-03-25T18:34:54.904 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:35:01.710 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:35:01.827 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.825+0000 7fe0880e2640 10 monclient: get_monmap_and_config 2024-03-25T18:35:01.827 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.825+0000 7fe0880e2640 10 monclient: build_initial_monmap 2024-03-25T18:35:01.827 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.826+0000 7fe0880e2640 10 monclient: monmap: 2024-03-25T18:35:01.827 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:35:01.827 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:35:01.827 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:35:01.826968+0000 2024-03-25T18:35:01.827 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:35:01.826968+0000 2024-03-25T18:35:01.828 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:01.828 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:35:01.828 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:35:01.828 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:35:01.828 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:35:01.828 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:35:01.830 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.826+0000 7fe0880e2640 1 Processor -- start 2024-03-25T18:35:01.830 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.826+0000 7fe0880e2640 1 -- start start 2024-03-25T18:35:01.830 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.826+0000 7fe0880e2640 10 monclient: init 2024-03-25T18:35:01.830 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.827+0000 7fe0880e2640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:01.830 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.827+0000 7fe0880e2640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:35:01.830 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.827+0000 7fe0880e2640 10 monclient(hunting): picked mon.noname-a con 0x7fe0800a98f0 addr v1:172.21.15.155:6789/0 2024-03-25T18:35:01.830 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.827+0000 7fe0880e2640 10 monclient(hunting): picked mon.noname-b con 0x7fe08013f710 addr v1:172.21.15.155:6790/0 2024-03-25T18:35:01.830 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.827+0000 7fe0880e2640 10 monclient(hunting): picked mon.noname-c con 0x7fe0801430c0 addr v1:172.21.15.167:6789/0 2024-03-25T18:35:01.830 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.827+0000 7fe0880e2640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe080142520 con 0x7fe0800a98f0 2024-03-25T18:35:01.830 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.827+0000 7fe0880e2640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0800b28a0 con 0x7fe0801430c0 2024-03-25T18:35:01.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.827+0000 7fe0880e2640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0800b2a30 con 0x7fe08013f710 2024-03-25T18:35:01.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.827+0000 7fe0880e2640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:01.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.827+0000 7fe0880e2640 10 monclient(hunting): authenticate will time out at 8148.137352s 2024-03-25T18:35:01.833 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.832+0000 7fe08667f640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7fe0801430c0 0x7fe08014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:42564/0 (socket says 172.21.15.155:42564) 2024-03-25T18:35:01.833 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.832+0000 7fe08667f640 1 -- 172.21.15.155:0/419800609 learned_addr learned my addr 172.21.15.155:0/419800609 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:35:01.841 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.840+0000 7fe084e7c640 1 -- 172.21.15.155:0/419800609 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3841134164 0 0) 0x7fe0800b2a30 con 0x7fe08013f710 2024-03-25T18:35:01.841 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.840+0000 7fe084e7c640 10 monclient(hunting): _negotiate 2024-03-25T18:35:01.841 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.840+0000 7fe084e7c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:01.841 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.840+0000 7fe084e7c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:01.841 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.840+0000 7fe084e7c640 10 monclient(hunting): my global_id is 4697 2024-03-25T18:35:01.842 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.840+0000 7fe084e7c640 1 -- 172.21.15.155:0/419800609 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe06c003520 con 0x7fe08013f710 2024-03-25T18:35:01.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.846+0000 7fe084e7c640 1 -- 172.21.15.155:0/419800609 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 837824857 0 0) 0x7fe0800b28a0 con 0x7fe0801430c0 2024-03-25T18:35:01.848 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.846+0000 7fe084e7c640 10 monclient(hunting): _negotiate 2024-03-25T18:35:01.848 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.846+0000 7fe084e7c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:01.848 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.846+0000 7fe084e7c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:01.848 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.846+0000 7fe084e7c640 10 monclient(hunting): my global_id is 4684 2024-03-25T18:35:01.848 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.846+0000 7fe084e7c640 1 -- 172.21.15.155:0/419800609 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0800b2a30 con 0x7fe0801430c0 2024-03-25T18:35:01.849 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.847+0000 7fe084e7c640 1 -- 172.21.15.155:0/419800609 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2475574392 0 0) 0x7fe06c003520 con 0x7fe08013f710 2024-03-25T18:35:01.849 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.847+0000 7fe084e7c640 1 -- 172.21.15.155:0/419800609 >> v1:172.21.15.167:6789/0 conn(0x7fe0801430c0 legacy=0x7fe08014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:01.849 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.847+0000 7fe084e7c640 1 -- 172.21.15.155:0/419800609 >> v1:172.21.15.155:6789/0 conn(0x7fe0800a98f0 legacy=0x7fe0800a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:35:01.849 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.847+0000 7fe084e7c640 10 monclient: _finish_hunting 0 2024-03-25T18:35:01.849 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.847+0000 7fe084e7c640 1 monclient: found mon.noname-b 2024-03-25T18:35:01.850 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.847+0000 7fe084e7c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:01.850 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.847+0000 7fe084e7c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-25T18:35:01.850 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.847+0000 7fe084e7c640 1 -- 172.21.15.155:0/419800609 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0800b44c0 con 0x7fe08013f710 2024-03-25T18:35:01.850 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.847+0000 7fe084e7c640 10 monclient: _finish_auth 0 2024-03-25T18:35:01.850 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.847+0000 7fe084e7c640 10 monclient: _check_auth_tickets 2024-03-25T18:35:01.850 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.847+0000 7fe084e7c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:01.850 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.847+0000 7fe084e7c640 1 -- 172.21.15.155:0/419800609 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fe070001690 con 0x7fe08013f710 2024-03-25T18:35:01.850 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.847+0000 7fe084e7c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:01.850 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.847+0000 7fe084e7c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:35:01.850 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.847+0000 7fe084e7c640 10 monclient: dump: 2024-03-25T18:35:01.850 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:35:01.850 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:35:01.851 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:01.851 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:01.851 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:01.851 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:35:01.851 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:35:01.851 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:35:01.851 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:35:01.851 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:35:01.851 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.847+0000 7fe0880e2640 5 monclient: authenticate success, global_id 4697 2024-03-25T18:35:01.851 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.847+0000 7fe0880e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:01.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.852+0000 7fe084e7c640 1 -- 172.21.15.155:0/419800609 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0700017f0 con 0x7fe08013f710 2024-03-25T18:35:01.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.852+0000 7fe084e7c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:01.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.852+0000 7fe0880e2640 10 monclient: get_monmap_and_config success 2024-03-25T18:35:01.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.852+0000 7fe0880e2640 4 set_mon_vals no callback set 2024-03-25T18:35:01.856 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.852+0000 7fe0880e2640 10 monclient: shutdown 2024-03-25T18:35:01.856 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.852+0000 7fe0880e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:01.856 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.852+0000 7fe0880e2640 1 -- 172.21.15.155:0/419800609 >> v1:172.21.15.155:6790/0 conn(0x7fe08013f710 legacy=0x7fe0801429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:01.856 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.853+0000 7fe0880e2640 1 -- 172.21.15.155:0/419800609 shutdown_connections 2024-03-25T18:35:01.856 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.853+0000 7fe0880e2640 1 -- 172.21.15.155:0/419800609 >> 172.21.15.155:0/419800609 conn(0x7fe0800b7ea0 msgr2=0x7fe0800b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:01.857 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.853+0000 7fe0880e2640 1 -- 172.21.15.155:0/419800609 shutdown_connections 2024-03-25T18:35:01.857 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.853+0000 7fe0880e2640 1 -- 172.21.15.155:0/419800609 wait complete. 2024-03-25T18:35:01.857 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.853+0000 7fe0880e2640 10 monclient: build_initial_monmap 2024-03-25T18:35:01.857 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.853+0000 7fe0880e2640 10 monclient: monmap: 2024-03-25T18:35:01.857 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:35:01.857 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:35:01.857 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:35:01.857 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:35:01.857 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:01.857 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:35:01.857 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:35:01.857 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:35:01.857 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:35:01.858 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:35:01.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.854+0000 7fe0880e2640 1 librados: starting msgr at 2024-03-25T18:35:01.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.854+0000 7fe0880e2640 1 librados: starting objecter 2024-03-25T18:35:01.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.854+0000 7fe0880e2640 1 Processor -- start 2024-03-25T18:35:01.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.854+0000 7fe0880e2640 1 -- start start 2024-03-25T18:35:01.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.854+0000 7fe0880e2640 1 librados: setting wanted keys 2024-03-25T18:35:01.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.854+0000 7fe0880e2640 1 librados: calling monclient init 2024-03-25T18:35:01.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.854+0000 7fe0880e2640 10 monclient: init 2024-03-25T18:35:01.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.854+0000 7fe0880e2640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:01.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.854+0000 7fe0880e2640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:35:01.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.854+0000 7fe0880e2640 10 monclient(hunting): picked mon.noname-b con 0x7fe0800a98f0 addr v1:172.21.15.167:6789/0 2024-03-25T18:35:01.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.854+0000 7fe0880e2640 10 monclient(hunting): picked mon.noname-c con 0x7fe08013f710 addr v1:172.21.15.155:6790/0 2024-03-25T18:35:01.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.854+0000 7fe0880e2640 10 monclient(hunting): picked mon.noname-a con 0x7fe0801430c0 addr v1:172.21.15.155:6789/0 2024-03-25T18:35:01.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.854+0000 7fe0880e2640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0800b0330 con 0x7fe0801430c0 2024-03-25T18:35:01.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.854+0000 7fe0880e2640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0800b0580 con 0x7fe0800a98f0 2024-03-25T18:35:01.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.854+0000 7fe0880e2640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0800b0860 con 0x7fe08013f710 2024-03-25T18:35:01.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.854+0000 7fe0880e2640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:01.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.854+0000 7fe0880e2640 10 monclient(hunting): authenticate will time out at 8148.164640s 2024-03-25T18:35:01.865 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.864+0000 7fe085e7e640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7fe0800a98f0 0x7fe0801d7c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:42598/0 (socket says 172.21.15.155:42598) 2024-03-25T18:35:01.865 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.864+0000 7fe085e7e640 1 -- 172.21.15.155:0/186909111 learned_addr learned my addr 172.21.15.155:0/186909111 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:35:01.865 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.864+0000 7fe0767fc640 10 client.?.objecter ms_handle_connect 0x7fe0800a98f0 2024-03-25T18:35:01.865 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.864+0000 7fe0767fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:01.869 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.868+0000 7fe0767fc640 10 client.?.objecter ms_handle_connect 0x7fe08013f710 2024-03-25T18:35:01.869 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.868+0000 7fe0767fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:01.879 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.878+0000 7fe0767fc640 10 client.?.objecter ms_handle_connect 0x7fe0801430c0 2024-03-25T18:35:01.879 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.878+0000 7fe0767fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:01.882 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.881+0000 7fe0767fc640 1 -- 172.21.15.155:0/186909111 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3120027939 0 0) 0x7fe0800b0860 con 0x7fe08013f710 2024-03-25T18:35:01.882 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.881+0000 7fe0767fc640 10 monclient(hunting): _negotiate 2024-03-25T18:35:01.882 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.881+0000 7fe0767fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:01.882 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.881+0000 7fe0767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:01.882 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.881+0000 7fe0767fc640 10 monclient(hunting): my global_id is 4700 2024-03-25T18:35:01.882 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.881+0000 7fe0767fc640 1 -- 172.21.15.155:0/186909111 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe064004eb0 con 0x7fe08013f710 2024-03-25T18:35:01.885 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.884+0000 7fe0767fc640 1 -- 172.21.15.155:0/186909111 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1032368813 0 0) 0x7fe0800b0580 con 0x7fe0800a98f0 2024-03-25T18:35:01.886 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.884+0000 7fe0767fc640 10 monclient(hunting): _negotiate 2024-03-25T18:35:01.886 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.884+0000 7fe0767fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:01.886 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.884+0000 7fe0767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:01.886 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.884+0000 7fe0767fc640 10 monclient(hunting): my global_id is 4687 2024-03-25T18:35:01.886 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.884+0000 7fe0767fc640 1 -- 172.21.15.155:0/186909111 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0800b0860 con 0x7fe0800a98f0 2024-03-25T18:35:01.889 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.888+0000 7fe0767fc640 1 -- 172.21.15.155:0/186909111 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1497446840 0 0) 0x7fe0800b0330 con 0x7fe0801430c0 2024-03-25T18:35:01.889 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.888+0000 7fe0767fc640 10 monclient(hunting): _negotiate 2024-03-25T18:35:01.889 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.888+0000 7fe0767fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:01.889 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.888+0000 7fe0767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:01.889 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.888+0000 7fe0767fc640 10 monclient(hunting): my global_id is 4689 2024-03-25T18:35:01.889 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.888+0000 7fe0767fc640 1 -- 172.21.15.155:0/186909111 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0800b0580 con 0x7fe0801430c0 2024-03-25T18:35:01.902 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.901+0000 7fe0767fc640 1 -- 172.21.15.155:0/186909111 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 408944510 0 0) 0x7fe0800b0580 con 0x7fe0801430c0 2024-03-25T18:35:01.902 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.901+0000 7fe0767fc640 1 -- 172.21.15.155:0/186909111 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0800b0330 con 0x7fe0801430c0 2024-03-25T18:35:01.902 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.901+0000 7fe0767fc640 1 -- 172.21.15.155:0/186909111 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fe078001b90 con 0x7fe0801430c0 2024-03-25T18:35:01.902 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.901+0000 7fe0767fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:01.902 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.901+0000 7fe0767fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:35:01.903 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.901+0000 7fe0767fc640 10 monclient(hunting): dump: 2024-03-25T18:35:01.903 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:35:01.903 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:35:01.903 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:01.903 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:01.903 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:01.903 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:35:01.903 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:35:01.903 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:35:01.903 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:35:01.903 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:35:01.903 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.901+0000 7fe0767fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:35:01.913 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.911+0000 7fe0767fc640 1 -- 172.21.15.155:0/186909111 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1857755473 0 0) 0x7fe064004eb0 con 0x7fe08013f710 2024-03-25T18:35:01.913 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.911+0000 7fe0767fc640 1 -- 172.21.15.155:0/186909111 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0800b0580 con 0x7fe08013f710 2024-03-25T18:35:01.913 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.911+0000 7fe0767fc640 1 -- 172.21.15.155:0/186909111 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fe070001ca0 con 0x7fe08013f710 2024-03-25T18:35:01.913 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.911+0000 7fe0767fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:01.913 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.911+0000 7fe0767fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:35:01.913 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.911+0000 7fe0767fc640 10 monclient(hunting): dump: 2024-03-25T18:35:01.913 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:35:01.913 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:35:01.913 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:01.913 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:01.914 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:01.914 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:35:01.914 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:35:01.914 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:35:01.914 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:35:01.914 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:35:01.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.912+0000 7fe0767fc640 1 -- 172.21.15.155:0/186909111 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2063853697 0 0) 0x7fe0800b0330 con 0x7fe0801430c0 2024-03-25T18:35:01.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.912+0000 7fe0767fc640 1 -- 172.21.15.155:0/186909111 >> v1:172.21.15.155:6790/0 conn(0x7fe08013f710 legacy=0x7fe0800ae4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:01.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.912+0000 7fe0767fc640 1 -- 172.21.15.155:0/186909111 >> v1:172.21.15.167:6789/0 conn(0x7fe0800a98f0 legacy=0x7fe0801d7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:01.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.912+0000 7fe0767fc640 10 monclient: _finish_hunting 0 2024-03-25T18:35:01.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.912+0000 7fe0767fc640 1 monclient: found mon.a 2024-03-25T18:35:01.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.912+0000 7fe0767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:01.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.912+0000 7fe0767fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:35:01.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.912+0000 7fe0767fc640 1 -- 172.21.15.155:0/186909111 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0801d7740 con 0x7fe0801430c0 2024-03-25T18:35:01.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.912+0000 7fe0767fc640 10 monclient: _finish_auth 0 2024-03-25T18:35:01.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.912+0000 7fe0767fc640 10 monclient: _check_auth_tickets 2024-03-25T18:35:01.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.912+0000 7fe0767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:01.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.912+0000 7fe0880e2640 5 monclient: authenticate success, global_id 4689 2024-03-25T18:35:01.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.912+0000 7fe0880e2640 10 monclient: _renew_subs 2024-03-25T18:35:01.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.912+0000 7fe0880e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:35:01.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.912+0000 7fe0880e2640 1 -- 172.21.15.155:0/186909111 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0801d72f0 con 0x7fe0801430c0 2024-03-25T18:35:01.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.912+0000 7fe0880e2640 10 client.4689.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:35:01.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.912+0000 7fe0880e2640 10 monclient: _renew_subs 2024-03-25T18:35:01.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.912+0000 7fe0880e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:35:01.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.912+0000 7fe0880e2640 1 -- 172.21.15.155:0/186909111 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0800b14b0 con 0x7fe0801430c0 2024-03-25T18:35:01.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.912+0000 7fe0880e2640 1 librados: init done 2024-03-25T18:35:01.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.913+0000 7fe0880e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:01.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.913+0000 7fe0880e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:01.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.913+0000 7fe0880e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:35:01.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:01.913+0000 7fe0880e2640 1 -- 172.21.15.155:0/186909111 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0801d78d0 con 0x7fe0801430c0 2024-03-25T18:35:02.228 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:02.226+0000 7fe0767fc640 1 -- 172.21.15.155:0/186909111 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe078001fa0 con 0x7fe0801430c0 2024-03-25T18:35:02.228 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:02.226+0000 7fe0767fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:02.228 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:02.226+0000 7fe0767fc640 1 -- 172.21.15.155:0/186909111 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fe078004d00 con 0x7fe0801430c0 2024-03-25T18:35:02.228 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:02.226+0000 7fe0767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:02.228 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:02.226+0000 7fe0767fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:35:02.228 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:02.226+0000 7fe0767fc640 10 monclient: dump: 2024-03-25T18:35:02.228 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:35:02.228 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:35:02.229 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:02.229 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:02.229 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:02.229 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:35:02.229 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:35:02.229 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:35:02.229 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:35:02.229 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:35:02.229 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:02.226+0000 7fe077fff640 4 set_mon_vals no callback set 2024-03-25T18:35:02.229 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:02.227+0000 7fe0767fc640 1 -- 172.21.15.155:0/186909111 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7fe07801acd0 con 0x7fe0801430c0 2024-03-25T18:35:02.230 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:02.227+0000 7fe085e7e640 10 monclient: get_auth_request con 0x7fe06406dec0 auth_method 0 2024-03-25T18:35:02.230 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:02.228+0000 7fe0767fc640 10 client.4689.objecter ms_handle_connect 0x7fe06406dec0 2024-03-25T18:35:02.234 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:02.233+0000 7fe0767fc640 1 -- 172.21.15.155:0/186909111 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1555029372 0 0) 0x7fe078054930 con 0x7fe0801430c0 2024-03-25T18:35:02.234 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:02.233+0000 7fe0767fc640 10 client.4689.objecter ms_dispatch 0x7fe080000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:35:02.234 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:02.233+0000 7fe0767fc640 3 client.4689.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:35:02.234 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:02.233+0000 7fe0767fc640 3 client.4689.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:35:02.234 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:02.233+0000 7fe0767fc640 20 client.4689.objecter dump_active .. 0 homeless 2024-03-25T18:35:02.234 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:02.233+0000 7fe0767fc640 1 -- 172.21.15.155:0/186909111 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe078050270 con 0x7fe0801430c0 2024-03-25T18:35:02.235 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:02.233+0000 7fe0767fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:02.235 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:02.233+0000 7fe0767fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:35:02.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:02.485+0000 7fe0880e2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:02.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:02.485+0000 7fe0880e2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:02.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:02.485+0000 7fe0880e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:35:02.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:02.485+0000 7fe0880e2640 1 -- 172.21.15.155:0/186909111 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe0800627b0 con 0x7fe0801430c0 2024-03-25T18:35:02.531 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:02.530+0000 7fe0767fc640 1 -- 172.21.15.155:0/186909111 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3170230695) 0x7fe078052370 con 0x7fe0801430c0 2024-03-25T18:35:02.532 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:02.530+0000 7fe0767fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:02.532 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:02.530+0000 7fe0767fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:35:02.532 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:35:02.532 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":11,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:34:47.433406+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:34:47.433406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:35:02.535 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:02.533+0000 7fe0880e2640 10 librados: watch_flush enter 2024-03-25T18:35:02.535 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:02.533+0000 7fe0880e2640 10 librados: watch_flush exit 2024-03-25T18:35:02.535 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:02.533+0000 7fe0880e2640 20 client.4689.objecter shutdown clearing up homeless session... 2024-03-25T18:35:02.535 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:02.533+0000 7fe0880e2640 10 client.4689.objecter successfully canceled tick 2024-03-25T18:35:02.535 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:02.533+0000 7fe0880e2640 1 -- 172.21.15.155:0/186909111 >> v1:172.21.15.167:6800/2919070503 conn(0x7fe06406dec0 legacy=0x7fe064070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:02.535 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:02.533+0000 7fe0880e2640 10 monclient: shutdown 2024-03-25T18:35:02.535 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:02.533+0000 7fe0880e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:02.535 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:02.533+0000 7fe0880e2640 1 -- 172.21.15.155:0/186909111 >> v1:172.21.15.155:6789/0 conn(0x7fe0801430c0 legacy=0x7fe0800af020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:02.536 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:02.533+0000 7fe0880e2640 1 -- 172.21.15.155:0/186909111 shutdown_connections 2024-03-25T18:35:02.536 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:02.533+0000 7fe0880e2640 1 -- 172.21.15.155:0/186909111 >> 172.21.15.155:0/186909111 conn(0x7fe0800b7ea0 msgr2=0x7fe080074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:02.536 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:02.533+0000 7fe0880e2640 1 -- 172.21.15.155:0/186909111 shutdown_connections 2024-03-25T18:35:02.536 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:02.533+0000 7fe0880e2640 1 -- 172.21.15.155:0/186909111 wait complete. 2024-03-25T18:35:02.536 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:02.533+0000 7fe0880e2640 1 librados: shutdown 2024-03-25T18:35:02.546 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:35:09.354 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:35:09.476 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.474+0000 7f7353ded640 10 monclient: get_monmap_and_config 2024-03-25T18:35:09.476 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.474+0000 7f7353ded640 10 monclient: build_initial_monmap 2024-03-25T18:35:09.476 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.474+0000 7f7353ded640 10 monclient: monmap: 2024-03-25T18:35:09.476 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:35:09.476 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:35:09.476 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:35:09.475239+0000 2024-03-25T18:35:09.476 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:35:09.475239+0000 2024-03-25T18:35:09.476 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:09.477 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:35:09.477 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:35:09.477 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:35:09.477 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:35:09.477 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:35:09.477 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.475+0000 7f7353ded640 1 Processor -- start 2024-03-25T18:35:09.477 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.475+0000 7f7353ded640 1 -- start start 2024-03-25T18:35:09.477 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.475+0000 7f7353ded640 10 monclient: init 2024-03-25T18:35:09.479 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.475+0000 7f7353ded640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:09.479 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.475+0000 7f7353ded640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:35:09.479 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.475+0000 7f7353ded640 10 monclient(hunting): picked mon.noname-b con 0x7f734c0b4d60 addr v1:172.21.15.155:6790/0 2024-03-25T18:35:09.479 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.475+0000 7f7353ded640 10 monclient(hunting): picked mon.noname-a con 0x7f734c0b1730 addr v1:172.21.15.155:6789/0 2024-03-25T18:35:09.479 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.475+0000 7f7353ded640 10 monclient(hunting): picked mon.noname-c con 0x7f734c14be90 addr v1:172.21.15.167:6789/0 2024-03-25T18:35:09.479 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.476+0000 7f7353ded640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f734c0af940 con 0x7f734c0b1730 2024-03-25T18:35:09.479 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.476+0000 7f7353ded640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f734c0b0a70 con 0x7f734c14be90 2024-03-25T18:35:09.480 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.476+0000 7f7353ded640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f734c0b0d50 con 0x7f734c0b4d60 2024-03-25T18:35:09.480 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.476+0000 7f7353ded640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:09.480 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.476+0000 7f7353ded640 10 monclient(hunting): authenticate will time out at 8155.785818s 2024-03-25T18:35:09.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.484+0000 7f7351388640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f734c0b1730 0x7f734c0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:56818/0 (socket says 172.21.15.155:56818) 2024-03-25T18:35:09.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.484+0000 7f7351388640 1 -- 172.21.15.155:0/3003289469 learned_addr learned my addr 172.21.15.155:0/3003289469 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:35:09.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.502+0000 7f7350b87640 1 -- 172.21.15.155:0/3003289469 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1701041445 0 0) 0x7f734c0af940 con 0x7f734c0b1730 2024-03-25T18:35:09.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.502+0000 7f7350b87640 10 monclient(hunting): _negotiate 2024-03-25T18:35:09.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.502+0000 7f7350b87640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:09.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.502+0000 7f7350b87640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:09.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.502+0000 7f7350b87640 10 monclient(hunting): my global_id is 4692 2024-03-25T18:35:09.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.502+0000 7f7350b87640 1 -- 172.21.15.155:0/3003289469 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7338003520 con 0x7f734c0b1730 2024-03-25T18:35:09.507 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.506+0000 7f7350b87640 1 -- 172.21.15.155:0/3003289469 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 682903157 0 0) 0x7f734c0b0a70 con 0x7f734c14be90 2024-03-25T18:35:09.507 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.506+0000 7f7350b87640 10 monclient(hunting): _negotiate 2024-03-25T18:35:09.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.506+0000 7f7350b87640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:09.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.506+0000 7f7350b87640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:09.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.506+0000 7f7350b87640 10 monclient(hunting): my global_id is 4690 2024-03-25T18:35:09.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.506+0000 7f7350b87640 1 -- 172.21.15.155:0/3003289469 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f734c0af940 con 0x7f734c14be90 2024-03-25T18:35:09.515 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.513+0000 7f7350b87640 1 -- 172.21.15.155:0/3003289469 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 772682866 0 0) 0x7f734c0b0d50 con 0x7f734c0b4d60 2024-03-25T18:35:09.515 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.513+0000 7f7350b87640 10 monclient(hunting): _negotiate 2024-03-25T18:35:09.515 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.513+0000 7f7350b87640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:09.515 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.513+0000 7f7350b87640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:09.515 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.513+0000 7f7350b87640 10 monclient(hunting): my global_id is 4703 2024-03-25T18:35:09.515 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.513+0000 7f7350b87640 1 -- 172.21.15.155:0/3003289469 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f734c0b0a70 con 0x7f734c0b4d60 2024-03-25T18:35:09.517 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.515+0000 7f7350b87640 1 -- 172.21.15.155:0/3003289469 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3039096497 0 0) 0x7f734c0af940 con 0x7f734c14be90 2024-03-25T18:35:09.517 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.515+0000 7f7350b87640 1 -- 172.21.15.155:0/3003289469 >> v1:172.21.15.155:6790/0 conn(0x7f734c0b4d60 legacy=0x7f734c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:09.517 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.515+0000 7f7350b87640 1 -- 172.21.15.155:0/3003289469 >> v1:172.21.15.155:6789/0 conn(0x7f734c0b1730 legacy=0x7f734c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:09.517 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.515+0000 7f7350b87640 10 monclient: _finish_hunting 0 2024-03-25T18:35:09.517 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.515+0000 7f7350b87640 1 monclient: found mon.noname-c 2024-03-25T18:35:09.517 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.515+0000 7f7350b87640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:09.517 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.515+0000 7f7350b87640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.167:6789/0 2024-03-25T18:35:09.518 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.515+0000 7f7350b87640 1 -- 172.21.15.155:0/3003289469 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f734c0b1030 con 0x7f734c14be90 2024-03-25T18:35:09.518 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.515+0000 7f7350b87640 10 monclient: _finish_auth 0 2024-03-25T18:35:09.518 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.515+0000 7f7350b87640 10 monclient: _check_auth_tickets 2024-03-25T18:35:09.518 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.515+0000 7f7350b87640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:09.518 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.515+0000 7f7353ded640 5 monclient: authenticate success, global_id 4690 2024-03-25T18:35:09.518 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.515+0000 7f7353ded640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:09.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.518+0000 7f7350b87640 1 -- 172.21.15.155:0/3003289469 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f73440019b0 con 0x7f734c14be90 2024-03-25T18:35:09.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.518+0000 7f7350b87640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:09.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.518+0000 7f7350b87640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:35:09.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.518+0000 7f7350b87640 10 monclient: dump: 2024-03-25T18:35:09.519 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:35:09.519 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:35:09.519 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:09.520 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:09.520 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:09.520 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:35:09.520 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:35:09.520 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:35:09.520 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:35:09.520 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:35:09.520 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.518+0000 7f7353ded640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:09.539 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.537+0000 7f7350b87640 1 -- 172.21.15.155:0/3003289469 <== mon.1 v1:172.21.15.167:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7344001c00 con 0x7f734c14be90 2024-03-25T18:35:09.539 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.537+0000 7f7350b87640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:09.540 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.537+0000 7f7350b87640 1 -- 172.21.15.155:0/3003289469 <== mon.1 v1:172.21.15.167:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f7344001f90 con 0x7f734c14be90 2024-03-25T18:35:09.540 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.537+0000 7f7353ded640 10 monclient: get_monmap_and_config success 2024-03-25T18:35:09.540 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.537+0000 7f7353ded640 4 set_mon_vals no callback set 2024-03-25T18:35:09.540 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.537+0000 7f7350b87640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:09.540 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.537+0000 7f7350b87640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:35:09.540 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.537+0000 7f7350b87640 10 monclient: dump: 2024-03-25T18:35:09.540 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:35:09.540 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:35:09.540 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:09.540 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:09.541 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:09.541 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:35:09.541 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:35:09.541 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:35:09.541 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:35:09.541 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:35:09.541 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.538+0000 7f7353ded640 10 monclient: shutdown 2024-03-25T18:35:09.541 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.538+0000 7f7353ded640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:09.541 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.538+0000 7f7353ded640 1 -- 172.21.15.155:0/3003289469 >> v1:172.21.15.167:6789/0 conn(0x7f734c14be90 legacy=0x7f734c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:09.541 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.538+0000 7f7353ded640 1 -- 172.21.15.155:0/3003289469 shutdown_connections 2024-03-25T18:35:09.541 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.538+0000 7f7353ded640 1 -- 172.21.15.155:0/3003289469 >> 172.21.15.155:0/3003289469 conn(0x7f734c0617a0 msgr2=0x7f734c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:09.542 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.538+0000 7f7353ded640 1 -- 172.21.15.155:0/3003289469 shutdown_connections 2024-03-25T18:35:09.542 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.538+0000 7f7353ded640 1 -- 172.21.15.155:0/3003289469 wait complete. 2024-03-25T18:35:09.543 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.539+0000 7f7353ded640 10 monclient: build_initial_monmap 2024-03-25T18:35:09.544 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.539+0000 7f7353ded640 10 monclient: monmap: 2024-03-25T18:35:09.544 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:35:09.544 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:35:09.544 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:35:09.544 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:35:09.544 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:09.544 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:35:09.544 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:35:09.544 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:35:09.545 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:35:09.545 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:35:09.545 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.539+0000 7f7353ded640 1 librados: starting msgr at 2024-03-25T18:35:09.545 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.539+0000 7f7353ded640 1 librados: starting objecter 2024-03-25T18:35:09.545 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.539+0000 7f7353ded640 1 Processor -- start 2024-03-25T18:35:09.545 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.540+0000 7f7353ded640 1 -- start start 2024-03-25T18:35:09.545 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.540+0000 7f7353ded640 1 librados: setting wanted keys 2024-03-25T18:35:09.545 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.540+0000 7f7353ded640 1 librados: calling monclient init 2024-03-25T18:35:09.545 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.540+0000 7f7353ded640 10 monclient: init 2024-03-25T18:35:09.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.540+0000 7f7353ded640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:09.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.540+0000 7f7353ded640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:35:09.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.540+0000 7f7353ded640 10 monclient(hunting): picked mon.noname-c con 0x7f734c0b1730 addr v1:172.21.15.155:6790/0 2024-03-25T18:35:09.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.540+0000 7f7353ded640 10 monclient(hunting): picked mon.noname-b con 0x7f734c0b4d60 addr v1:172.21.15.167:6789/0 2024-03-25T18:35:09.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.540+0000 7f7353ded640 10 monclient(hunting): picked mon.noname-a con 0x7f734c14be90 addr v1:172.21.15.155:6789/0 2024-03-25T18:35:09.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.540+0000 7f7353ded640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f734c138dd0 con 0x7f734c14be90 2024-03-25T18:35:09.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.540+0000 7f7353ded640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f734c0aa740 con 0x7f734c0b4d60 2024-03-25T18:35:09.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.540+0000 7f7353ded640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f734c0aaa20 con 0x7f734c0b1730 2024-03-25T18:35:09.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.540+0000 7f7353ded640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:09.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.540+0000 7f7353ded640 10 monclient(hunting): authenticate will time out at 8155.850517s 2024-03-25T18:35:09.553 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.552+0000 7f735238a640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f734c14be90 0x7f734c1e31b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:56854/0 (socket says 172.21.15.155:56854) 2024-03-25T18:35:09.553 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.552+0000 7f735238a640 1 -- 172.21.15.155:0/623244317 learned_addr learned my addr 172.21.15.155:0/623244317 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:35:09.554 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.553+0000 7f73427fc640 10 client.?.objecter ms_handle_connect 0x7f734c14be90 2024-03-25T18:35:09.554 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.553+0000 7f73427fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:09.557 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.556+0000 7f73427fc640 10 client.?.objecter ms_handle_connect 0x7f734c0b4d60 2024-03-25T18:35:09.557 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.556+0000 7f73427fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:09.562 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.561+0000 7f73427fc640 10 client.?.objecter ms_handle_connect 0x7f734c0b1730 2024-03-25T18:35:09.563 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.561+0000 7f73427fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:09.564 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.563+0000 7f73427fc640 1 -- 172.21.15.155:0/623244317 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2264953796 0 0) 0x7f734c138dd0 con 0x7f734c14be90 2024-03-25T18:35:09.564 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.563+0000 7f73427fc640 10 monclient(hunting): _negotiate 2024-03-25T18:35:09.564 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.563+0000 7f73427fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:09.564 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.563+0000 7f73427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:09.564 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.563+0000 7f73427fc640 10 monclient(hunting): my global_id is 4695 2024-03-25T18:35:09.565 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.563+0000 7f73427fc640 1 -- 172.21.15.155:0/623244317 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7330004ee0 con 0x7f734c14be90 2024-03-25T18:35:09.576 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.575+0000 7f73427fc640 1 -- 172.21.15.155:0/623244317 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 903217348 0 0) 0x7f734c0aa740 con 0x7f734c0b4d60 2024-03-25T18:35:09.577 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.575+0000 7f73427fc640 10 monclient(hunting): _negotiate 2024-03-25T18:35:09.577 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.575+0000 7f73427fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:09.577 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.575+0000 7f73427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:09.577 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.575+0000 7f73427fc640 10 monclient(hunting): my global_id is 4693 2024-03-25T18:35:09.577 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.575+0000 7f73427fc640 1 -- 172.21.15.155:0/623244317 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f734c138dd0 con 0x7f734c0b4d60 2024-03-25T18:35:09.577 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.575+0000 7f73427fc640 1 -- 172.21.15.155:0/623244317 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1933845491 0 0) 0x7f7330004ee0 con 0x7f734c14be90 2024-03-25T18:35:09.577 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.575+0000 7f73427fc640 1 -- 172.21.15.155:0/623244317 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f734c0aa740 con 0x7f734c14be90 2024-03-25T18:35:09.577 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.575+0000 7f73427fc640 1 -- 172.21.15.155:0/623244317 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f7344001f50 con 0x7f734c14be90 2024-03-25T18:35:09.577 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.575+0000 7f73427fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:09.577 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.575+0000 7f73427fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:35:09.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.575+0000 7f73427fc640 10 monclient(hunting): dump: 2024-03-25T18:35:09.578 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:35:09.578 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:35:09.578 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:09.578 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:09.578 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:09.578 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:35:09.578 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:35:09.578 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:35:09.579 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:35:09.579 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:35:09.579 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.575+0000 7f73427fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:35:09.579 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.578+0000 7f73427fc640 1 -- 172.21.15.155:0/623244317 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1232534100 0 0) 0x7f734c0aaa20 con 0x7f734c0b1730 2024-03-25T18:35:09.580 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.578+0000 7f73427fc640 10 monclient(hunting): _negotiate 2024-03-25T18:35:09.580 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.578+0000 7f73427fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:09.580 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.578+0000 7f73427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:09.580 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.578+0000 7f73427fc640 10 monclient(hunting): my global_id is 4706 2024-03-25T18:35:09.580 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.578+0000 7f73427fc640 1 -- 172.21.15.155:0/623244317 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7330004ee0 con 0x7f734c0b1730 2024-03-25T18:35:09.590 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.589+0000 7f73427fc640 1 -- 172.21.15.155:0/623244317 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3071653690 0 0) 0x7f734c0aa740 con 0x7f734c14be90 2024-03-25T18:35:09.591 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.589+0000 7f73427fc640 1 -- 172.21.15.155:0/623244317 >> v1:172.21.15.155:6790/0 conn(0x7f734c0b1730 legacy=0x7f734c137ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:09.591 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.589+0000 7f73427fc640 1 -- 172.21.15.155:0/623244317 >> v1:172.21.15.167:6789/0 conn(0x7f734c0b4d60 legacy=0x7f734c1d7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:09.591 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.589+0000 7f73427fc640 10 monclient: _finish_hunting 0 2024-03-25T18:35:09.591 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.589+0000 7f73427fc640 1 monclient: found mon.a 2024-03-25T18:35:09.591 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.589+0000 7f73427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:09.591 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.589+0000 7f73427fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:35:09.591 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.589+0000 7f73427fc640 1 -- 172.21.15.155:0/623244317 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f734c0aad00 con 0x7f734c14be90 2024-03-25T18:35:09.591 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.589+0000 7f73427fc640 10 monclient: _finish_auth 0 2024-03-25T18:35:09.591 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.589+0000 7f73427fc640 10 monclient: _check_auth_tickets 2024-03-25T18:35:09.592 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.589+0000 7f73427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:09.592 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.589+0000 7f7353ded640 5 monclient: authenticate success, global_id 4695 2024-03-25T18:35:09.592 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.589+0000 7f7353ded640 10 monclient: _renew_subs 2024-03-25T18:35:09.592 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.589+0000 7f7353ded640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:35:09.592 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.589+0000 7f7353ded640 1 -- 172.21.15.155:0/623244317 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f734c1385f0 con 0x7f734c14be90 2024-03-25T18:35:09.592 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.589+0000 7f7353ded640 10 client.4695.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:35:09.592 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.589+0000 7f7353ded640 10 monclient: _renew_subs 2024-03-25T18:35:09.592 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.589+0000 7f7353ded640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:35:09.592 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.589+0000 7f7353ded640 1 -- 172.21.15.155:0/623244317 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f734c13bbd0 con 0x7f734c14be90 2024-03-25T18:35:09.592 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.589+0000 7f7353ded640 1 librados: init done 2024-03-25T18:35:09.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.590+0000 7f7353ded640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:09.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.590+0000 7f7353ded640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:09.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.590+0000 7f7353ded640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:35:09.594 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.590+0000 7f7353ded640 1 -- 172.21.15.155:0/623244317 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f734c1388f0 con 0x7f734c14be90 2024-03-25T18:35:09.886 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.884+0000 7f73427fc640 1 -- 172.21.15.155:0/623244317 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7344003c20 con 0x7f734c14be90 2024-03-25T18:35:09.886 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.884+0000 7f73427fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:09.886 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.884+0000 7f73427fc640 1 -- 172.21.15.155:0/623244317 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f7344003ff0 con 0x7f734c14be90 2024-03-25T18:35:09.886 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.884+0000 7f73427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:09.886 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.884+0000 7f73427fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:35:09.886 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.884+0000 7f73427fc640 10 monclient: dump: 2024-03-25T18:35:09.887 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:35:09.887 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:35:09.887 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:09.887 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:09.887 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:09.887 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:35:09.887 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:35:09.887 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:35:09.887 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:35:09.887 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:35:09.887 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.884+0000 7f7343fff640 4 set_mon_vals no callback set 2024-03-25T18:35:09.888 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.885+0000 7f73427fc640 1 -- 172.21.15.155:0/623244317 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7f7344004210 con 0x7f734c14be90 2024-03-25T18:35:09.888 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.886+0000 7f7351b89640 10 monclient: get_auth_request con 0x7f733006de50 auth_method 0 2024-03-25T18:35:09.889 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.886+0000 7f73427fc640 10 client.4695.objecter ms_handle_connect 0x7f733006de50 2024-03-25T18:35:09.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.891+0000 7f73427fc640 1 -- 172.21.15.155:0/623244317 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1555029372 0 0) 0x7f7344054b40 con 0x7f734c14be90 2024-03-25T18:35:09.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.891+0000 7f73427fc640 10 client.4695.objecter ms_dispatch 0x7f734c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:35:09.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.891+0000 7f73427fc640 3 client.4695.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:35:09.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.891+0000 7f73427fc640 3 client.4695.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:35:09.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.891+0000 7f73427fc640 20 client.4695.objecter dump_active .. 0 homeless 2024-03-25T18:35:09.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.891+0000 7f73427fc640 1 -- 172.21.15.155:0/623244317 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7344050650 con 0x7f734c14be90 2024-03-25T18:35:09.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.891+0000 7f73427fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:09.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:09.891+0000 7f73427fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:35:10.147 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:10.146+0000 7f7353ded640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:10.147 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:10.146+0000 7f7353ded640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:10.147 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:10.146+0000 7f7353ded640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:35:10.147 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:10.146+0000 7f7353ded640 1 -- 172.21.15.155:0/623244317 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f734c0627b0 con 0x7f734c14be90 2024-03-25T18:35:10.179 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:10.178+0000 7f73427fc640 1 -- 172.21.15.155:0/623244317 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3170230695) 0x7f7344052580 con 0x7f734c14be90 2024-03-25T18:35:10.179 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:10.178+0000 7f73427fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:10.180 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:10.178+0000 7f73427fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:35:10.180 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:35:10.180 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":11,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:34:47.433406+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:34:47.433406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:35:10.183 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:10.181+0000 7f7353ded640 10 librados: watch_flush enter 2024-03-25T18:35:10.183 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:10.181+0000 7f7353ded640 10 librados: watch_flush exit 2024-03-25T18:35:10.183 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:10.181+0000 7f7353ded640 20 client.4695.objecter shutdown clearing up homeless session... 2024-03-25T18:35:10.183 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:10.181+0000 7f7353ded640 10 client.4695.objecter successfully canceled tick 2024-03-25T18:35:10.183 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:10.181+0000 7f7353ded640 1 -- 172.21.15.155:0/623244317 >> v1:172.21.15.167:6800/2919070503 conn(0x7f733006de50 legacy=0x7f7330070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:10.183 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:10.181+0000 7f7353ded640 10 monclient: shutdown 2024-03-25T18:35:10.183 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:10.181+0000 7f7353ded640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:10.183 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:10.181+0000 7f7353ded640 1 -- 172.21.15.155:0/623244317 >> v1:172.21.15.155:6789/0 conn(0x7f734c14be90 legacy=0x7f734c1e31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:10.183 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:10.181+0000 7f7353ded640 1 -- 172.21.15.155:0/623244317 shutdown_connections 2024-03-25T18:35:10.183 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:10.181+0000 7f7353ded640 1 -- 172.21.15.155:0/623244317 >> 172.21.15.155:0/623244317 conn(0x7f734c0617a0 msgr2=0x7f734c0b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:10.184 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:10.181+0000 7f7353ded640 1 -- 172.21.15.155:0/623244317 shutdown_connections 2024-03-25T18:35:10.184 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:10.181+0000 7f7353ded640 1 -- 172.21.15.155:0/623244317 wait complete. 2024-03-25T18:35:10.184 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:10.181+0000 7f7353ded640 1 librados: shutdown 2024-03-25T18:35:10.194 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:35:12.621 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:10:46.226 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:12.671 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:10:46.275 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:12.701 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:10:46.305 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:12.737 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:10:46.341 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:12.740 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:10:46.318 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:12.742 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:10:46.321 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:12.750 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:10:46.355 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:12.778 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:10:46.357 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:12.792 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:10:46.397 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:17.000 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:35:17.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.113+0000 7f0d50964640 10 monclient: get_monmap_and_config 2024-03-25T18:35:17.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.113+0000 7f0d50964640 10 monclient: build_initial_monmap 2024-03-25T18:35:17.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.113+0000 7f0d50964640 10 monclient: monmap: 2024-03-25T18:35:17.115 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:35:17.115 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:35:17.115 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:35:17.114986+0000 2024-03-25T18:35:17.115 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:35:17.114986+0000 2024-03-25T18:35:17.115 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:17.115 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:35:17.115 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:35:17.115 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:35:17.116 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:35:17.116 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:35:17.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.113+0000 7f0d50964640 1 Processor -- start 2024-03-25T18:35:17.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.113+0000 7f0d50964640 1 -- start start 2024-03-25T18:35:17.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.113+0000 7f0d50964640 10 monclient: init 2024-03-25T18:35:17.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.114+0000 7f0d50964640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:17.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.114+0000 7f0d50964640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:35:17.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.114+0000 7f0d50964640 10 monclient(hunting): picked mon.noname-c con 0x7f0d4c0b4d60 addr v1:172.21.15.167:6789/0 2024-03-25T18:35:17.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.114+0000 7f0d50964640 10 monclient(hunting): picked mon.noname-b con 0x7f0d4c0b1730 addr v1:172.21.15.155:6790/0 2024-03-25T18:35:17.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.114+0000 7f0d50964640 10 monclient(hunting): picked mon.noname-a con 0x7f0d4c14be90 addr v1:172.21.15.155:6789/0 2024-03-25T18:35:17.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.114+0000 7f0d50964640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d4c0af940 con 0x7f0d4c14be90 2024-03-25T18:35:17.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.114+0000 7f0d50964640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d4c0b0a70 con 0x7f0d4c0b4d60 2024-03-25T18:35:17.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.114+0000 7f0d50964640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d4c0b0d50 con 0x7f0d4c0b1730 2024-03-25T18:35:17.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.114+0000 7f0d50964640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:17.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.114+0000 7f0d50964640 10 monclient(hunting): authenticate will time out at 8163.425587s 2024-03-25T18:35:17.126 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.124+0000 7f0d4ad9d640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f0d4c14be90 0x7f0d4c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:34204/0 (socket says 172.21.15.155:34204) 2024-03-25T18:35:17.126 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.124+0000 7f0d4ad9d640 1 -- 172.21.15.155:0/3688050945 learned_addr learned my addr 172.21.15.155:0/3688050945 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:35:17.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.136+0000 7f0d4959a640 1 -- 172.21.15.155:0/3688050945 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4123499348 0 0) 0x7f0d4c0b0a70 con 0x7f0d4c0b4d60 2024-03-25T18:35:17.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.136+0000 7f0d4959a640 10 monclient(hunting): _negotiate 2024-03-25T18:35:17.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.136+0000 7f0d4959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:17.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.136+0000 7f0d4959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:17.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.136+0000 7f0d4959a640 10 monclient(hunting): my global_id is 4696 2024-03-25T18:35:17.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.136+0000 7f0d4959a640 1 -- 172.21.15.155:0/3688050945 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d34003520 con 0x7f0d4c0b4d60 2024-03-25T18:35:17.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.139+0000 7f0d4959a640 1 -- 172.21.15.155:0/3688050945 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1945066862 0 0) 0x7f0d4c0af940 con 0x7f0d4c14be90 2024-03-25T18:35:17.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.139+0000 7f0d4959a640 10 monclient(hunting): _negotiate 2024-03-25T18:35:17.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.139+0000 7f0d4959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:17.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.139+0000 7f0d4959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:17.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.139+0000 7f0d4959a640 10 monclient(hunting): my global_id is 4698 2024-03-25T18:35:17.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.139+0000 7f0d4959a640 1 -- 172.21.15.155:0/3688050945 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d4c0b0a70 con 0x7f0d4c14be90 2024-03-25T18:35:17.151 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.149+0000 7f0d4959a640 1 -- 172.21.15.155:0/3688050945 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2458367199 0 0) 0x7f0d4c0b0d50 con 0x7f0d4c0b1730 2024-03-25T18:35:17.151 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.149+0000 7f0d4959a640 10 monclient(hunting): _negotiate 2024-03-25T18:35:17.151 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.149+0000 7f0d4959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:17.151 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.149+0000 7f0d4959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:17.152 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.149+0000 7f0d4959a640 10 monclient(hunting): my global_id is 4709 2024-03-25T18:35:17.152 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.149+0000 7f0d4959a640 1 -- 172.21.15.155:0/3688050945 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d4c0af940 con 0x7f0d4c0b1730 2024-03-25T18:35:17.157 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.155+0000 7f0d4959a640 1 -- 172.21.15.155:0/3688050945 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1835914635 0 0) 0x7f0d4c0b0a70 con 0x7f0d4c14be90 2024-03-25T18:35:17.157 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.155+0000 7f0d4959a640 1 -- 172.21.15.155:0/3688050945 >> v1:172.21.15.155:6790/0 conn(0x7f0d4c0b1730 legacy=0x7f0d4c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:17.157 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.155+0000 7f0d4959a640 1 -- 172.21.15.155:0/3688050945 >> v1:172.21.15.167:6789/0 conn(0x7f0d4c0b4d60 legacy=0x7f0d4c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:17.157 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.155+0000 7f0d4959a640 10 monclient: _finish_hunting 0 2024-03-25T18:35:17.157 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.155+0000 7f0d4959a640 1 monclient: found mon.noname-a 2024-03-25T18:35:17.157 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.155+0000 7f0d4959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:17.157 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.155+0000 7f0d4959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-25T18:35:17.157 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.155+0000 7f0d4959a640 1 -- 172.21.15.155:0/3688050945 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d4c0b1030 con 0x7f0d4c14be90 2024-03-25T18:35:17.157 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.155+0000 7f0d4959a640 10 monclient: _finish_auth 0 2024-03-25T18:35:17.157 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.155+0000 7f0d4959a640 10 monclient: _check_auth_tickets 2024-03-25T18:35:17.158 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.155+0000 7f0d4959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:17.158 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.155+0000 7f0d4959a640 1 -- 172.21.15.155:0/3688050945 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f0d440019f0 con 0x7f0d4c14be90 2024-03-25T18:35:17.158 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.155+0000 7f0d4959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:17.158 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.155+0000 7f0d4959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:35:17.158 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.155+0000 7f0d4959a640 10 monclient: dump: 2024-03-25T18:35:17.158 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:35:17.158 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:35:17.158 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:17.158 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:17.158 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:17.158 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:35:17.158 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:35:17.158 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:35:17.158 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:35:17.158 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:35:17.159 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.155+0000 7f0d50964640 5 monclient: authenticate success, global_id 4698 2024-03-25T18:35:17.159 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.155+0000 7f0d50964640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:17.168 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.166+0000 7f0d4959a640 1 -- 172.21.15.155:0/3688050945 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d44001b50 con 0x7f0d4c14be90 2024-03-25T18:35:17.168 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.166+0000 7f0d4959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:17.168 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.166+0000 7f0d4959a640 1 -- 172.21.15.155:0/3688050945 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f0d44001f60 con 0x7f0d4c14be90 2024-03-25T18:35:17.169 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.166+0000 7f0d50964640 10 monclient: get_monmap_and_config success 2024-03-25T18:35:17.169 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.166+0000 7f0d50964640 4 set_mon_vals no callback set 2024-03-25T18:35:17.169 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.166+0000 7f0d4959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:17.169 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.166+0000 7f0d4959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:35:17.169 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.166+0000 7f0d4959a640 10 monclient: dump: 2024-03-25T18:35:17.169 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:35:17.169 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:35:17.169 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:17.169 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:17.169 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:17.169 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:35:17.169 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:35:17.169 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:35:17.169 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:35:17.169 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:35:17.171 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.167+0000 7f0d50964640 10 monclient: shutdown 2024-03-25T18:35:17.171 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.167+0000 7f0d50964640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:17.171 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.167+0000 7f0d50964640 1 -- 172.21.15.155:0/3688050945 >> v1:172.21.15.155:6789/0 conn(0x7f0d4c14be90 legacy=0x7f0d4c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:17.171 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.167+0000 7f0d50964640 1 -- 172.21.15.155:0/3688050945 shutdown_connections 2024-03-25T18:35:17.171 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.167+0000 7f0d50964640 1 -- 172.21.15.155:0/3688050945 >> 172.21.15.155:0/3688050945 conn(0x7f0d4c0617a0 msgr2=0x7f0d4c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:17.171 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.167+0000 7f0d50964640 1 -- 172.21.15.155:0/3688050945 shutdown_connections 2024-03-25T18:35:17.171 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.167+0000 7f0d50964640 1 -- 172.21.15.155:0/3688050945 wait complete. 2024-03-25T18:35:17.171 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.168+0000 7f0d50964640 10 monclient: build_initial_monmap 2024-03-25T18:35:17.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.168+0000 7f0d50964640 10 monclient: monmap: 2024-03-25T18:35:17.172 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:35:17.172 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:35:17.172 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:35:17.172 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:35:17.172 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:17.172 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:35:17.172 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:35:17.172 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:35:17.172 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:35:17.172 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:35:17.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.169+0000 7f0d50964640 1 librados: starting msgr at 2024-03-25T18:35:17.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.169+0000 7f0d50964640 1 librados: starting objecter 2024-03-25T18:35:17.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.169+0000 7f0d50964640 1 Processor -- start 2024-03-25T18:35:17.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.169+0000 7f0d50964640 1 -- start start 2024-03-25T18:35:17.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.169+0000 7f0d50964640 1 librados: setting wanted keys 2024-03-25T18:35:17.173 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.169+0000 7f0d50964640 1 librados: calling monclient init 2024-03-25T18:35:17.173 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.169+0000 7f0d50964640 10 monclient: init 2024-03-25T18:35:17.174 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.170+0000 7f0d50964640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:17.174 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.170+0000 7f0d50964640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:35:17.174 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.170+0000 7f0d50964640 10 monclient(hunting): picked mon.noname-b con 0x7f0d4c0b1730 addr v1:172.21.15.167:6789/0 2024-03-25T18:35:17.174 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.170+0000 7f0d50964640 10 monclient(hunting): picked mon.noname-a con 0x7f0d4c0b4d60 addr v1:172.21.15.155:6789/0 2024-03-25T18:35:17.174 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.170+0000 7f0d50964640 10 monclient(hunting): picked mon.noname-c con 0x7f0d4c14be90 addr v1:172.21.15.155:6790/0 2024-03-25T18:35:17.174 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.170+0000 7f0d50964640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d4c1d7ab0 con 0x7f0d4c0b4d60 2024-03-25T18:35:17.174 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.170+0000 7f0d50964640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d4c0aac90 con 0x7f0d4c0b1730 2024-03-25T18:35:17.174 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.170+0000 7f0d50964640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d4c0a9b80 con 0x7f0d4c14be90 2024-03-25T18:35:17.174 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.170+0000 7f0d50964640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:17.174 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.170+0000 7f0d50964640 10 monclient(hunting): authenticate will time out at 8163.481072s 2024-03-25T18:35:17.189 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.187+0000 7f0d4ad9d640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f0d4c14be90 0x7f0d4c1e22e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:39470/0 (socket says 172.21.15.155:39470) 2024-03-25T18:35:17.190 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.187+0000 7f0d4ad9d640 1 -- 172.21.15.155:0/2650729132 learned_addr learned my addr 172.21.15.155:0/2650729132 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:35:17.192 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.190+0000 7f0d3affd640 10 client.?.objecter ms_handle_connect 0x7f0d4c14be90 2024-03-25T18:35:17.192 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.190+0000 7f0d3affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:17.193 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.191+0000 7f0d3affd640 10 client.?.objecter ms_handle_connect 0x7f0d4c0b1730 2024-03-25T18:35:17.193 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.191+0000 7f0d3affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:17.205 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.203+0000 7f0d3affd640 1 -- 172.21.15.155:0/2650729132 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1533465757 0 0) 0x7f0d4c0aac90 con 0x7f0d4c0b1730 2024-03-25T18:35:17.205 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.203+0000 7f0d3affd640 10 monclient(hunting): _negotiate 2024-03-25T18:35:17.205 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.203+0000 7f0d3affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:17.205 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.203+0000 7f0d3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:17.205 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.203+0000 7f0d3affd640 10 monclient(hunting): my global_id is 4699 2024-03-25T18:35:17.205 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.203+0000 7f0d3affd640 1 -- 172.21.15.155:0/2650729132 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d2c004eb0 con 0x7f0d4c0b1730 2024-03-25T18:35:17.206 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.203+0000 7f0d3affd640 1 -- 172.21.15.155:0/2650729132 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2316987522 0 0) 0x7f0d4c0a9b80 con 0x7f0d4c14be90 2024-03-25T18:35:17.206 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.203+0000 7f0d3affd640 10 monclient(hunting): _negotiate 2024-03-25T18:35:17.206 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.203+0000 7f0d3affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:17.206 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.203+0000 7f0d3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:17.206 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.203+0000 7f0d3affd640 10 monclient(hunting): my global_id is 4712 2024-03-25T18:35:17.206 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.203+0000 7f0d3affd640 1 -- 172.21.15.155:0/2650729132 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d4c0aac90 con 0x7f0d4c14be90 2024-03-25T18:35:17.222 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.219+0000 7f0d3affd640 1 -- 172.21.15.155:0/2650729132 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1582049748 0 0) 0x7f0d4c0aac90 con 0x7f0d4c14be90 2024-03-25T18:35:17.222 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.219+0000 7f0d3affd640 1 -- 172.21.15.155:0/2650729132 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d4c0a9b80 con 0x7f0d4c14be90 2024-03-25T18:35:17.222 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.219+0000 7f0d3affd640 1 -- 172.21.15.155:0/2650729132 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f0d44001f20 con 0x7f0d4c14be90 2024-03-25T18:35:17.222 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.219+0000 7f0d3affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:17.222 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.219+0000 7f0d3affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:35:17.222 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.219+0000 7f0d3affd640 10 monclient(hunting): dump: 2024-03-25T18:35:17.222 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:35:17.222 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:35:17.222 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:17.222 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:17.223 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:17.223 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:35:17.223 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:35:17.223 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:35:17.223 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:35:17.223 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:35:17.223 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.219+0000 7f0d3affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:35:17.223 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.220+0000 7f0d3affd640 1 -- 172.21.15.155:0/2650729132 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3313406383 0 0) 0x7f0d2c004eb0 con 0x7f0d4c0b1730 2024-03-25T18:35:17.223 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.220+0000 7f0d3affd640 1 -- 172.21.15.155:0/2650729132 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d4c0aac90 con 0x7f0d4c0b1730 2024-03-25T18:35:17.223 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.220+0000 7f0d3affd640 1 -- 172.21.15.155:0/2650729132 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f0d3c001950 con 0x7f0d4c0b1730 2024-03-25T18:35:17.223 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.220+0000 7f0d3affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:17.223 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.220+0000 7f0d3affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:35:17.223 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.220+0000 7f0d3affd640 10 monclient(hunting): dump: 2024-03-25T18:35:17.223 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:35:17.223 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:35:17.224 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:17.224 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:17.224 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:17.224 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:35:17.224 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:35:17.224 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:35:17.224 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:35:17.224 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:35:17.224 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.222+0000 7f0d3affd640 10 client.?.objecter ms_handle_connect 0x7f0d4c0b4d60 2024-03-25T18:35:17.224 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.222+0000 7f0d3affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:17.235 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.233+0000 7f0d3affd640 1 -- 172.21.15.155:0/2650729132 <== mon.1 v1:172.21.15.167:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 110389175 0 0) 0x7f0d4c0aac90 con 0x7f0d4c0b1730 2024-03-25T18:35:17.235 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.233+0000 7f0d3affd640 1 -- 172.21.15.155:0/2650729132 >> v1:172.21.15.155:6790/0 conn(0x7f0d4c14be90 legacy=0x7f0d4c1e22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:17.235 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.233+0000 7f0d3affd640 1 -- 172.21.15.155:0/2650729132 >> v1:172.21.15.155:6789/0 conn(0x7f0d4c0b4d60 legacy=0x7f0d4c1d7f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:17.235 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.233+0000 7f0d3affd640 10 monclient: _finish_hunting 0 2024-03-25T18:35:17.235 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.233+0000 7f0d3affd640 1 monclient: found mon.b 2024-03-25T18:35:17.235 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.233+0000 7f0d3affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:17.236 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.233+0000 7f0d3affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:35:17.236 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.233+0000 7f0d3affd640 1 -- 172.21.15.155:0/2650729132 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d4c1d8650 con 0x7f0d4c0b1730 2024-03-25T18:35:17.236 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.233+0000 7f0d3affd640 10 monclient: _finish_auth 0 2024-03-25T18:35:17.236 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.233+0000 7f0d3affd640 10 monclient: _check_auth_tickets 2024-03-25T18:35:17.236 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.233+0000 7f0d3affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:17.236 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.233+0000 7f0d50964640 5 monclient: authenticate success, global_id 4699 2024-03-25T18:35:17.236 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.233+0000 7f0d50964640 10 monclient: _renew_subs 2024-03-25T18:35:17.236 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.233+0000 7f0d50964640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:35:17.236 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.233+0000 7f0d50964640 1 -- 172.21.15.155:0/2650729132 --> v1:172.21.15.167:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d4c140380 con 0x7f0d4c0b1730 2024-03-25T18:35:17.236 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.233+0000 7f0d50964640 10 client.4699.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:35:17.236 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.233+0000 7f0d50964640 10 monclient: _renew_subs 2024-03-25T18:35:17.236 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.233+0000 7f0d50964640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:35:17.236 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.233+0000 7f0d50964640 1 -- 172.21.15.155:0/2650729132 --> v1:172.21.15.167:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d4c1434f0 con 0x7f0d4c0b1730 2024-03-25T18:35:17.236 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.233+0000 7f0d50964640 1 librados: init done 2024-03-25T18:35:17.237 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.234+0000 7f0d50964640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:17.237 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.234+0000 7f0d50964640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:17.237 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.234+0000 7f0d50964640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:35:17.237 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.234+0000 7f0d50964640 1 -- 172.21.15.155:0/2650729132 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d4c0a9d10 con 0x7f0d4c0b1730 2024-03-25T18:35:17.502 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.500+0000 7f0d3affd640 1 -- 172.21.15.155:0/2650729132 <== mon.1 v1:172.21.15.167:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d3c003610 con 0x7f0d4c0b1730 2024-03-25T18:35:17.502 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.500+0000 7f0d3affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:17.502 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.500+0000 7f0d3affd640 1 -- 172.21.15.155:0/2650729132 <== mon.1 v1:172.21.15.167:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f0d3c001d80 con 0x7f0d4c0b1730 2024-03-25T18:35:17.502 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.500+0000 7f0d3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:17.502 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.500+0000 7f0d3affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:35:17.502 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.500+0000 7f0d3affd640 10 monclient: dump: 2024-03-25T18:35:17.503 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:35:17.503 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:35:17.503 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:17.503 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:17.503 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:17.503 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:35:17.503 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:35:17.503 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:35:17.503 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:35:17.503 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:35:17.503 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.500+0000 7f0d48d99640 4 set_mon_vals no callback set 2024-03-25T18:35:17.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.501+0000 7f0d3affd640 1 -- 172.21.15.155:0/2650729132 <== mon.1 v1:172.21.15.167:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7f0d3c019810 con 0x7f0d4c0b1730 2024-03-25T18:35:17.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.501+0000 7f0d49d9b640 10 monclient: get_auth_request con 0x7f0d2c06dec0 auth_method 0 2024-03-25T18:35:17.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.501+0000 7f0d3affd640 10 client.4699.objecter ms_handle_connect 0x7f0d2c06dec0 2024-03-25T18:35:17.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.507+0000 7f0d3affd640 1 -- 172.21.15.155:0/2650729132 <== mon.1 v1:172.21.15.167:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1555029372 0 0) 0x7f0d3c053580 con 0x7f0d4c0b1730 2024-03-25T18:35:17.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.507+0000 7f0d3affd640 10 client.4699.objecter ms_dispatch 0x7f0d4c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:35:17.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.507+0000 7f0d3affd640 3 client.4699.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:35:17.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.507+0000 7f0d3affd640 3 client.4699.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:35:17.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.507+0000 7f0d3affd640 20 client.4699.objecter dump_active .. 0 homeless 2024-03-25T18:35:17.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.508+0000 7f0d3affd640 1 -- 172.21.15.155:0/2650729132 <== mon.1 v1:172.21.15.167:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0d3c04eec0 con 0x7f0d4c0b1730 2024-03-25T18:35:17.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.508+0000 7f0d3affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:17.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.508+0000 7f0d3affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:35:17.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.717+0000 7f0d50964640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:17.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.717+0000 7f0d50964640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:17.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.717+0000 7f0d50964640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:35:17.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.717+0000 7f0d50964640 1 -- 172.21.15.155:0/2650729132 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0d4c1d87e0 con 0x7f0d4c0b1730 2024-03-25T18:35:17.759 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.757+0000 7f0d3affd640 1 -- 172.21.15.155:0/2650729132 <== mon.1 v1:172.21.15.167:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3170230695) 0x7f0d3c050fc0 con 0x7f0d4c0b1730 2024-03-25T18:35:17.759 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.757+0000 7f0d3affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:17.759 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.757+0000 7f0d3affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:35:17.760 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:35:17.760 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":11,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:34:47.433406+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:34:47.433406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:35:17.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.759+0000 7f0d50964640 10 librados: watch_flush enter 2024-03-25T18:35:17.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.759+0000 7f0d50964640 10 librados: watch_flush exit 2024-03-25T18:35:17.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.759+0000 7f0d50964640 20 client.4699.objecter shutdown clearing up homeless session... 2024-03-25T18:35:17.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.759+0000 7f0d50964640 10 client.4699.objecter successfully canceled tick 2024-03-25T18:35:17.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.760+0000 7f0d50964640 1 -- 172.21.15.155:0/2650729132 >> v1:172.21.15.167:6800/2919070503 conn(0x7f0d2c06dec0 legacy=0x7f0d2c070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:17.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.760+0000 7f0d50964640 10 monclient: shutdown 2024-03-25T18:35:17.763 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.760+0000 7f0d50964640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:17.763 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.760+0000 7f0d50964640 1 -- 172.21.15.155:0/2650729132 >> v1:172.21.15.167:6789/0 conn(0x7f0d4c0b1730 legacy=0x7f0d4c0a9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:17.763 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.760+0000 7f0d50964640 1 -- 172.21.15.155:0/2650729132 shutdown_connections 2024-03-25T18:35:17.763 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.760+0000 7f0d50964640 1 -- 172.21.15.155:0/2650729132 >> 172.21.15.155:0/2650729132 conn(0x7f0d4c0617a0 msgr2=0x7f0d4c13c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:17.763 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.760+0000 7f0d50964640 1 -- 172.21.15.155:0/2650729132 shutdown_connections 2024-03-25T18:35:17.763 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.760+0000 7f0d50964640 1 -- 172.21.15.155:0/2650729132 wait complete. 2024-03-25T18:35:17.763 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:17.760+0000 7f0d50964640 1 librados: shutdown 2024-03-25T18:35:17.775 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:35:24.581 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:35:24.699 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.698+0000 7f8cbf3cf640 10 monclient: get_monmap_and_config 2024-03-25T18:35:24.699 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.698+0000 7f8cbf3cf640 10 monclient: build_initial_monmap 2024-03-25T18:35:24.700 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.698+0000 7f8cbf3cf640 10 monclient: monmap: 2024-03-25T18:35:24.700 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:35:24.700 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:35:24.700 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:35:24.699477+0000 2024-03-25T18:35:24.700 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:35:24.699477+0000 2024-03-25T18:35:24.700 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:24.700 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:35:24.700 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:35:24.700 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:35:24.700 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:35:24.700 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:35:24.701 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.699+0000 7f8cbf3cf640 1 Processor -- start 2024-03-25T18:35:24.701 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.699+0000 7f8cbf3cf640 1 -- start start 2024-03-25T18:35:24.702 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.699+0000 7f8cbf3cf640 10 monclient: init 2024-03-25T18:35:24.702 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.700+0000 7f8cbf3cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:24.702 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.700+0000 7f8cbf3cf640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:35:24.702 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.700+0000 7f8cbf3cf640 10 monclient(hunting): picked mon.noname-c con 0x7f8cb80a98f0 addr v1:172.21.15.167:6789/0 2024-03-25T18:35:24.702 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.700+0000 7f8cbf3cf640 10 monclient(hunting): picked mon.noname-a con 0x7f8cb813f710 addr v1:172.21.15.155:6789/0 2024-03-25T18:35:24.702 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.700+0000 7f8cbf3cf640 10 monclient(hunting): picked mon.noname-b con 0x7f8cb81430c0 addr v1:172.21.15.155:6790/0 2024-03-25T18:35:24.702 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.700+0000 7f8cbf3cf640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cb8142520 con 0x7f8cb813f710 2024-03-25T18:35:24.702 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.700+0000 7f8cbf3cf640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cb80b28a0 con 0x7f8cb80a98f0 2024-03-25T18:35:24.702 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.700+0000 7f8cbf3cf640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cb80b2a30 con 0x7f8cb81430c0 2024-03-25T18:35:24.702 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.700+0000 7f8cbf3cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:24.702 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.700+0000 7f8cbf3cf640 10 monclient(hunting): authenticate will time out at 8171.010145s 2024-03-25T18:35:24.710 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.709+0000 7f8cbc96a640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f8cb813f710 0x7f8cb81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:39030/0 (socket says 172.21.15.155:39030) 2024-03-25T18:35:24.710 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.709+0000 7f8cbc96a640 1 -- 172.21.15.155:0/2851163934 learned_addr learned my addr 172.21.15.155:0/2851163934 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:35:24.729 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.728+0000 7f8caffff640 1 -- 172.21.15.155:0/2851163934 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2830923687 0 0) 0x7f8cb8142520 con 0x7f8cb813f710 2024-03-25T18:35:24.729 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.728+0000 7f8caffff640 10 monclient(hunting): _negotiate 2024-03-25T18:35:24.729 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.728+0000 7f8caffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:24.729 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.728+0000 7f8caffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:24.729 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.728+0000 7f8caffff640 10 monclient(hunting): my global_id is 4704 2024-03-25T18:35:24.729 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.728+0000 7f8caffff640 1 -- 172.21.15.155:0/2851163934 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ca4003520 con 0x7f8cb813f710 2024-03-25T18:35:24.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.730+0000 7f8caffff640 1 -- 172.21.15.155:0/2851163934 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4033067859 0 0) 0x7f8cb80b2a30 con 0x7f8cb81430c0 2024-03-25T18:35:24.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.730+0000 7f8caffff640 10 monclient(hunting): _negotiate 2024-03-25T18:35:24.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.730+0000 7f8caffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:24.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.730+0000 7f8caffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:24.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.730+0000 7f8caffff640 10 monclient(hunting): my global_id is 4715 2024-03-25T18:35:24.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.730+0000 7f8caffff640 1 -- 172.21.15.155:0/2851163934 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cb8142520 con 0x7f8cb81430c0 2024-03-25T18:35:24.737 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.736+0000 7f8caffff640 1 -- 172.21.15.155:0/2851163934 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1825211467 0 0) 0x7f8ca4003520 con 0x7f8cb813f710 2024-03-25T18:35:24.737 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.736+0000 7f8caffff640 1 -- 172.21.15.155:0/2851163934 >> v1:172.21.15.155:6790/0 conn(0x7f8cb81430c0 legacy=0x7f8cb814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:24.737 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.736+0000 7f8caffff640 1 -- 172.21.15.155:0/2851163934 >> v1:172.21.15.167:6789/0 conn(0x7f8cb80a98f0 legacy=0x7f8cb80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:24.737 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.736+0000 7f8caffff640 10 monclient: _finish_hunting 0 2024-03-25T18:35:24.737 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.736+0000 7f8caffff640 1 monclient: found mon.noname-a 2024-03-25T18:35:24.737 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.736+0000 7f8caffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:24.737 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.736+0000 7f8caffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-25T18:35:24.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.736+0000 7f8caffff640 1 -- 172.21.15.155:0/2851163934 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cb80b44c0 con 0x7f8cb813f710 2024-03-25T18:35:24.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.736+0000 7f8caffff640 10 monclient: _finish_auth 0 2024-03-25T18:35:24.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.736+0000 7f8caffff640 10 monclient: _check_auth_tickets 2024-03-25T18:35:24.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.736+0000 7f8caffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:24.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.736+0000 7f8caffff640 1 -- 172.21.15.155:0/2851163934 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f8ca8001690 con 0x7f8cb813f710 2024-03-25T18:35:24.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.736+0000 7f8caffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:24.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.736+0000 7f8caffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:35:24.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.736+0000 7f8caffff640 10 monclient: dump: 2024-03-25T18:35:24.738 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:35:24.738 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:35:24.738 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:24.738 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:24.738 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:24.738 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:35:24.739 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:35:24.739 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:35:24.739 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:35:24.739 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:35:24.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.736+0000 7f8cbf3cf640 5 monclient: authenticate success, global_id 4704 2024-03-25T18:35:24.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.736+0000 7f8cbf3cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:24.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.740+0000 7f8caffff640 1 -- 172.21.15.155:0/2851163934 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ca80017f0 con 0x7f8cb813f710 2024-03-25T18:35:24.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.740+0000 7f8caffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:24.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.740+0000 7f8caffff640 1 -- 172.21.15.155:0/2851163934 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f8ca8001c00 con 0x7f8cb813f710 2024-03-25T18:35:24.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.740+0000 7f8cbf3cf640 10 monclient: get_monmap_and_config success 2024-03-25T18:35:24.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.740+0000 7f8cbf3cf640 4 set_mon_vals no callback set 2024-03-25T18:35:24.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.740+0000 7f8caffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:24.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.740+0000 7f8caffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:35:24.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.740+0000 7f8caffff640 10 monclient: dump: 2024-03-25T18:35:24.742 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:35:24.742 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:35:24.742 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:24.742 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:24.742 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:24.742 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:35:24.742 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:35:24.742 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:35:24.742 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:35:24.743 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:35:24.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.741+0000 7f8cbf3cf640 10 monclient: shutdown 2024-03-25T18:35:24.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.741+0000 7f8cbf3cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:24.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.741+0000 7f8cbf3cf640 1 -- 172.21.15.155:0/2851163934 >> v1:172.21.15.155:6789/0 conn(0x7f8cb813f710 legacy=0x7f8cb81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:24.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.741+0000 7f8cbf3cf640 1 -- 172.21.15.155:0/2851163934 shutdown_connections 2024-03-25T18:35:24.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.741+0000 7f8cbf3cf640 1 -- 172.21.15.155:0/2851163934 >> 172.21.15.155:0/2851163934 conn(0x7f8cb80b7ea0 msgr2=0x7f8cb80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:24.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.741+0000 7f8cbf3cf640 1 -- 172.21.15.155:0/2851163934 shutdown_connections 2024-03-25T18:35:24.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.741+0000 7f8cbf3cf640 1 -- 172.21.15.155:0/2851163934 wait complete. 2024-03-25T18:35:24.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.742+0000 7f8cbf3cf640 10 monclient: build_initial_monmap 2024-03-25T18:35:24.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.742+0000 7f8cbf3cf640 10 monclient: monmap: 2024-03-25T18:35:24.745 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:35:24.746 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:35:24.746 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:35:24.746 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:35:24.746 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:24.746 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:35:24.746 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:35:24.746 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:35:24.746 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:35:24.746 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:35:24.746 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.742+0000 7f8cbf3cf640 1 librados: starting msgr at 2024-03-25T18:35:24.746 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.742+0000 7f8cbf3cf640 1 librados: starting objecter 2024-03-25T18:35:24.746 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.742+0000 7f8cbf3cf640 1 Processor -- start 2024-03-25T18:35:24.746 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.742+0000 7f8cbf3cf640 1 -- start start 2024-03-25T18:35:24.747 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.742+0000 7f8cbf3cf640 1 librados: setting wanted keys 2024-03-25T18:35:24.747 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.742+0000 7f8cbf3cf640 1 librados: calling monclient init 2024-03-25T18:35:24.747 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.742+0000 7f8cbf3cf640 10 monclient: init 2024-03-25T18:35:24.747 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.743+0000 7f8cbf3cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:24.747 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.743+0000 7f8cbf3cf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:35:24.747 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.743+0000 7f8cbf3cf640 10 monclient(hunting): picked mon.noname-c con 0x7f8cb80a98f0 addr v1:172.21.15.155:6790/0 2024-03-25T18:35:24.747 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.743+0000 7f8cbf3cf640 10 monclient(hunting): picked mon.noname-b con 0x7f8cb813f710 addr v1:172.21.15.167:6789/0 2024-03-25T18:35:24.747 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.743+0000 7f8cbf3cf640 10 monclient(hunting): picked mon.noname-a con 0x7f8cb81430c0 addr v1:172.21.15.155:6789/0 2024-03-25T18:35:24.747 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.743+0000 7f8cbf3cf640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cb81d82d0 con 0x7f8cb81430c0 2024-03-25T18:35:24.747 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.743+0000 7f8cbf3cf640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cb80abb90 con 0x7f8cb813f710 2024-03-25T18:35:24.747 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.743+0000 7f8cbf3cf640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cb81d76a0 con 0x7f8cb80a98f0 2024-03-25T18:35:24.747 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.743+0000 7f8cbf3cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:24.747 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.743+0000 7f8cbf3cf640 10 monclient(hunting): authenticate will time out at 8171.053099s 2024-03-25T18:35:24.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.750+0000 7f8cbd16b640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f8cb80a98f0 0x7f8cb81385d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:43968/0 (socket says 172.21.15.155:43968) 2024-03-25T18:35:24.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.750+0000 7f8cbd16b640 1 -- 172.21.15.155:0/960561959 learned_addr learned my addr 172.21.15.155:0/960561959 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:35:24.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.751+0000 7f8cadffb640 10 client.?.objecter ms_handle_connect 0x7f8cb80a98f0 2024-03-25T18:35:24.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.751+0000 7f8cadffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:24.763 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.762+0000 7f8cadffb640 1 -- 172.21.15.155:0/960561959 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 72523204 0 0) 0x7f8cb81d76a0 con 0x7f8cb80a98f0 2024-03-25T18:35:24.763 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.762+0000 7f8cadffb640 10 monclient(hunting): _negotiate 2024-03-25T18:35:24.763 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.762+0000 7f8cadffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:24.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.762+0000 7f8cadffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:24.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.762+0000 7f8cadffb640 10 monclient(hunting): my global_id is 4718 2024-03-25T18:35:24.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.762+0000 7f8cadffb640 1 -- 172.21.15.155:0/960561959 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c9c004eb0 con 0x7f8cb80a98f0 2024-03-25T18:35:24.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.766+0000 7f8cadffb640 10 client.?.objecter ms_handle_connect 0x7f8cb813f710 2024-03-25T18:35:24.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.766+0000 7f8cadffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:24.772 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.771+0000 7f8cadffb640 1 -- 172.21.15.155:0/960561959 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1268482378 0 0) 0x7f8c9c004eb0 con 0x7f8cb80a98f0 2024-03-25T18:35:24.772 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.771+0000 7f8cadffb640 1 -- 172.21.15.155:0/960561959 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8cb81d76a0 con 0x7f8cb80a98f0 2024-03-25T18:35:24.773 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.771+0000 7f8cadffb640 1 -- 172.21.15.155:0/960561959 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f8cb4001db0 con 0x7f8cb80a98f0 2024-03-25T18:35:24.773 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.771+0000 7f8cadffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:24.773 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.771+0000 7f8cadffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:35:24.773 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.771+0000 7f8cadffb640 10 monclient(hunting): dump: 2024-03-25T18:35:24.773 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:35:24.773 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:35:24.773 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:24.773 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:24.773 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:24.773 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:35:24.773 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:35:24.773 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:35:24.773 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:35:24.773 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:35:24.774 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.771+0000 7f8cadffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:35:24.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.777+0000 7f8cadffb640 10 client.?.objecter ms_handle_connect 0x7f8cb81430c0 2024-03-25T18:35:24.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.777+0000 7f8cadffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:24.782 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.781+0000 7f8cadffb640 1 -- 172.21.15.155:0/960561959 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2505418619 0 0) 0x7f8cb80abb90 con 0x7f8cb813f710 2024-03-25T18:35:24.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.781+0000 7f8cadffb640 10 monclient(hunting): _negotiate 2024-03-25T18:35:24.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.781+0000 7f8cadffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:24.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.781+0000 7f8cadffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:24.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.781+0000 7f8cadffb640 10 monclient(hunting): my global_id is 4705 2024-03-25T18:35:24.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.781+0000 7f8cadffb640 1 -- 172.21.15.155:0/960561959 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c9c004eb0 con 0x7f8cb813f710 2024-03-25T18:35:24.794 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.793+0000 7f8cadffb640 1 -- 172.21.15.155:0/960561959 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4202375304 0 0) 0x7f8cb81d82d0 con 0x7f8cb81430c0 2024-03-25T18:35:24.794 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.793+0000 7f8cadffb640 10 monclient(hunting): _negotiate 2024-03-25T18:35:24.794 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.793+0000 7f8cadffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:24.795 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.793+0000 7f8cadffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:24.795 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.793+0000 7f8cadffb640 10 monclient(hunting): my global_id is 4707 2024-03-25T18:35:24.795 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.793+0000 7f8cadffb640 1 -- 172.21.15.155:0/960561959 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cb80abb90 con 0x7f8cb81430c0 2024-03-25T18:35:24.797 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.796+0000 7f8cadffb640 1 -- 172.21.15.155:0/960561959 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3840642319 0 0) 0x7f8cb81d76a0 con 0x7f8cb80a98f0 2024-03-25T18:35:24.797 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.796+0000 7f8cadffb640 1 -- 172.21.15.155:0/960561959 >> v1:172.21.15.167:6789/0 conn(0x7f8cb813f710 legacy=0x7f8cb81d7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:24.797 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.796+0000 7f8cadffb640 1 -- 172.21.15.155:0/960561959 >> v1:172.21.15.155:6789/0 conn(0x7f8cb81430c0 legacy=0x7f8cb8134010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:24.798 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.796+0000 7f8cadffb640 10 monclient: _finish_hunting 0 2024-03-25T18:35:24.798 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.796+0000 7f8cadffb640 1 monclient: found mon.c 2024-03-25T18:35:24.798 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.796+0000 7f8cadffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:24.798 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.796+0000 7f8cadffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:35:24.798 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.796+0000 7f8cadffb640 1 -- 172.21.15.155:0/960561959 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cb8134720 con 0x7f8cb80a98f0 2024-03-25T18:35:24.798 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.796+0000 7f8cadffb640 10 monclient: _finish_auth 0 2024-03-25T18:35:24.798 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.796+0000 7f8cadffb640 10 monclient: _check_auth_tickets 2024-03-25T18:35:24.798 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.796+0000 7f8cadffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:24.798 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.796+0000 7f8cbf3cf640 5 monclient: authenticate success, global_id 4718 2024-03-25T18:35:24.798 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.796+0000 7f8cbf3cf640 10 monclient: _renew_subs 2024-03-25T18:35:24.798 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.796+0000 7f8cbf3cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:35:24.798 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.796+0000 7f8cbf3cf640 1 -- 172.21.15.155:0/960561959 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cb8134970 con 0x7f8cb80a98f0 2024-03-25T18:35:24.798 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.796+0000 7f8cbf3cf640 10 client.4718.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:35:24.798 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.796+0000 7f8cbf3cf640 10 monclient: _renew_subs 2024-03-25T18:35:24.799 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.796+0000 7f8cbf3cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:35:24.799 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.796+0000 7f8cbf3cf640 1 -- 172.21.15.155:0/960561959 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cb8137a10 con 0x7f8cb80a98f0 2024-03-25T18:35:24.799 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.796+0000 7f8cbf3cf640 1 librados: init done 2024-03-25T18:35:24.799 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.797+0000 7f8cbf3cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:24.799 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.797+0000 7f8cbf3cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:24.799 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.797+0000 7f8cbf3cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:35:24.800 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:24.797+0000 7f8cbf3cf640 1 -- 172.21.15.155:0/960561959 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cb81d7830 con 0x7f8cb80a98f0 2024-03-25T18:35:25.102 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:25.100+0000 7f8cadffb640 1 -- 172.21.15.155:0/960561959 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8cb4002070 con 0x7f8cb80a98f0 2024-03-25T18:35:25.102 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:25.100+0000 7f8cadffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:25.102 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:25.100+0000 7f8cadffb640 1 -- 172.21.15.155:0/960561959 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f8cb4003de0 con 0x7f8cb80a98f0 2024-03-25T18:35:25.102 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:25.100+0000 7f8cadffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:25.102 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:25.101+0000 7f8cadffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:35:25.102 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:25.101+0000 7f8cadffb640 10 monclient: dump: 2024-03-25T18:35:25.102 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:35:25.103 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:35:25.103 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:25.103 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:25.103 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:25.103 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:35:25.103 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:35:25.103 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:35:25.103 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:35:25.103 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:35:25.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:25.100+0000 7f8caf7fe640 4 set_mon_vals no callback set 2024-03-25T18:35:25.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:25.101+0000 7f8cadffb640 1 -- 172.21.15.155:0/960561959 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7f8cb4004000 con 0x7f8cb80a98f0 2024-03-25T18:35:25.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:25.102+0000 7f8cbc96a640 10 monclient: get_auth_request con 0x7f8c9c06de10 auth_method 0 2024-03-25T18:35:25.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:25.102+0000 7f8cadffb640 10 client.4718.objecter ms_handle_connect 0x7f8c9c06de10 2024-03-25T18:35:25.108 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:25.107+0000 7f8cadffb640 1 -- 172.21.15.155:0/960561959 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1555029372 0 0) 0x7f8cb4054ca0 con 0x7f8cb80a98f0 2024-03-25T18:35:25.108 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:25.107+0000 7f8cadffb640 10 client.4718.objecter ms_dispatch 0x7f8cb8000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:35:25.109 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:25.107+0000 7f8cadffb640 3 client.4718.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:35:25.109 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:25.107+0000 7f8cadffb640 3 client.4718.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:35:25.109 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:25.107+0000 7f8cadffb640 20 client.4718.objecter dump_active .. 0 homeless 2024-03-25T18:35:25.109 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:25.107+0000 7f8cadffb640 1 -- 172.21.15.155:0/960561959 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8cb40507b0 con 0x7f8cb80a98f0 2024-03-25T18:35:25.109 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:25.107+0000 7f8cadffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:25.109 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:25.107+0000 7f8cadffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:35:25.334 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:25.333+0000 7f8cbf3cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:25.334 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:25.333+0000 7f8cbf3cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:25.334 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:25.333+0000 7f8cbf3cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:35:25.334 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:25.333+0000 7f8cbf3cf640 1 -- 172.21.15.155:0/960561959 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8cb81d7490 con 0x7f8cb80a98f0 2024-03-25T18:35:25.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:25.440+0000 7f8cadffb640 1 -- 172.21.15.155:0/960561959 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3170230695) 0x7f8cb40526e0 con 0x7f8cb80a98f0 2024-03-25T18:35:25.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:25.440+0000 7f8cadffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:25.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:25.440+0000 7f8cadffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:35:25.442 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:35:25.442 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":11,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:34:47.433406+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:34:47.433406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:35:25.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:25.443+0000 7f8cbf3cf640 10 librados: watch_flush enter 2024-03-25T18:35:25.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:25.443+0000 7f8cbf3cf640 10 librados: watch_flush exit 2024-03-25T18:35:25.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:25.443+0000 7f8cbf3cf640 20 client.4718.objecter shutdown clearing up homeless session... 2024-03-25T18:35:25.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:25.443+0000 7f8cbf3cf640 10 client.4718.objecter successfully canceled tick 2024-03-25T18:35:25.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:25.443+0000 7f8cbf3cf640 1 -- 172.21.15.155:0/960561959 >> v1:172.21.15.167:6800/2919070503 conn(0x7f8c9c06de10 legacy=0x7f8c9c0702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:25.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:25.443+0000 7f8cbf3cf640 10 monclient: shutdown 2024-03-25T18:35:25.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:25.443+0000 7f8cbf3cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:25.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:25.443+0000 7f8cbf3cf640 1 -- 172.21.15.155:0/960561959 >> v1:172.21.15.155:6790/0 conn(0x7f8cb80a98f0 legacy=0x7f8cb81385d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:25.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:25.443+0000 7f8cbf3cf640 1 -- 172.21.15.155:0/960561959 shutdown_connections 2024-03-25T18:35:25.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:25.443+0000 7f8cbf3cf640 1 -- 172.21.15.155:0/960561959 >> 172.21.15.155:0/960561959 conn(0x7f8cb80b7ea0 msgr2=0x7f8cb8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:25.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:25.443+0000 7f8cbf3cf640 1 -- 172.21.15.155:0/960561959 shutdown_connections 2024-03-25T18:35:25.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:25.443+0000 7f8cbf3cf640 1 -- 172.21.15.155:0/960561959 wait complete. 2024-03-25T18:35:25.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:25.443+0000 7f8cbf3cf640 1 librados: shutdown 2024-03-25T18:35:25.456 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:35:32.263 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:35:32.380 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.379+0000 7fc81fe05640 10 monclient: get_monmap_and_config 2024-03-25T18:35:32.380 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.379+0000 7fc81fe05640 10 monclient: build_initial_monmap 2024-03-25T18:35:32.381 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.379+0000 7fc81fe05640 10 monclient: monmap: 2024-03-25T18:35:32.381 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:35:32.381 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:35:32.381 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:35:32.380844+0000 2024-03-25T18:35:32.381 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:35:32.380844+0000 2024-03-25T18:35:32.381 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:32.381 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:35:32.381 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:35:32.381 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:35:32.381 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:35:32.381 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:35:32.382 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.380+0000 7fc81fe05640 1 Processor -- start 2024-03-25T18:35:32.382 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.380+0000 7fc81fe05640 1 -- start start 2024-03-25T18:35:32.383 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.380+0000 7fc81fe05640 10 monclient: init 2024-03-25T18:35:32.383 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.381+0000 7fc81fe05640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:32.383 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.381+0000 7fc81fe05640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:35:32.383 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.381+0000 7fc81fe05640 10 monclient(hunting): picked mon.noname-b con 0x7fc8180a98f0 addr v1:172.21.15.155:6790/0 2024-03-25T18:35:32.383 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.381+0000 7fc81fe05640 10 monclient(hunting): picked mon.noname-c con 0x7fc81813f710 addr v1:172.21.15.167:6789/0 2024-03-25T18:35:32.383 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.381+0000 7fc81fe05640 10 monclient(hunting): picked mon.noname-a con 0x7fc8181430c0 addr v1:172.21.15.155:6789/0 2024-03-25T18:35:32.383 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.381+0000 7fc81fe05640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc818142520 con 0x7fc8181430c0 2024-03-25T18:35:32.383 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.381+0000 7fc81fe05640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8180b28a0 con 0x7fc81813f710 2024-03-25T18:35:32.383 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.381+0000 7fc81fe05640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8180b2a30 con 0x7fc8180a98f0 2024-03-25T18:35:32.383 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.381+0000 7fc81fe05640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:32.383 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.381+0000 7fc81fe05640 10 monclient(hunting): authenticate will time out at 8178.691527s 2024-03-25T18:35:32.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.389+0000 7fc81e3a2640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fc8181430c0 0x7fc81814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:59976/0 (socket says 172.21.15.155:59976) 2024-03-25T18:35:32.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.389+0000 7fc81e3a2640 1 -- 172.21.15.155:0/3637729889 learned_addr learned my addr 172.21.15.155:0/3637729889 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:35:32.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.398+0000 7fc81cb9f640 1 -- 172.21.15.155:0/3637729889 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3005124683 0 0) 0x7fc818142520 con 0x7fc8181430c0 2024-03-25T18:35:32.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.398+0000 7fc81cb9f640 10 monclient(hunting): _negotiate 2024-03-25T18:35:32.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.398+0000 7fc81cb9f640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:32.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.398+0000 7fc81cb9f640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:32.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.398+0000 7fc81cb9f640 10 monclient(hunting): my global_id is 4710 2024-03-25T18:35:32.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.398+0000 7fc81cb9f640 1 -- 172.21.15.155:0/3637729889 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc804003520 con 0x7fc8181430c0 2024-03-25T18:35:32.401 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.400+0000 7fc81cb9f640 1 -- 172.21.15.155:0/3637729889 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3594127074 0 0) 0x7fc8180b2a30 con 0x7fc8180a98f0 2024-03-25T18:35:32.401 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.400+0000 7fc81cb9f640 10 monclient(hunting): _negotiate 2024-03-25T18:35:32.402 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.400+0000 7fc81cb9f640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:32.402 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.400+0000 7fc81cb9f640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:32.402 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.400+0000 7fc81cb9f640 10 monclient(hunting): my global_id is 4721 2024-03-25T18:35:32.402 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.400+0000 7fc81cb9f640 1 -- 172.21.15.155:0/3637729889 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc818142520 con 0x7fc8180a98f0 2024-03-25T18:35:32.414 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.412+0000 7fc81cb9f640 1 -- 172.21.15.155:0/3637729889 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2118147387 0 0) 0x7fc804003520 con 0x7fc8181430c0 2024-03-25T18:35:32.414 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.412+0000 7fc81cb9f640 1 -- 172.21.15.155:0/3637729889 >> v1:172.21.15.155:6790/0 conn(0x7fc8180a98f0 legacy=0x7fc8180a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:32.414 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.412+0000 7fc81cb9f640 1 -- 172.21.15.155:0/3637729889 >> v1:172.21.15.167:6789/0 conn(0x7fc81813f710 legacy=0x7fc8181429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:32.414 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.412+0000 7fc81cb9f640 10 monclient: _finish_hunting 0 2024-03-25T18:35:32.414 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.412+0000 7fc81cb9f640 1 monclient: found mon.noname-a 2024-03-25T18:35:32.414 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.412+0000 7fc81cb9f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:32.414 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.412+0000 7fc81cb9f640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-25T18:35:32.414 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.412+0000 7fc81cb9f640 1 -- 172.21.15.155:0/3637729889 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8180b44c0 con 0x7fc8181430c0 2024-03-25T18:35:32.414 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.412+0000 7fc81cb9f640 10 monclient: _finish_auth 0 2024-03-25T18:35:32.414 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.412+0000 7fc81cb9f640 10 monclient: _check_auth_tickets 2024-03-25T18:35:32.414 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.412+0000 7fc81cb9f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:32.414 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.412+0000 7fc81cb9f640 1 -- 172.21.15.155:0/3637729889 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fc810001a10 con 0x7fc8181430c0 2024-03-25T18:35:32.415 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.412+0000 7fc81fe05640 5 monclient: authenticate success, global_id 4710 2024-03-25T18:35:32.415 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.412+0000 7fc81fe05640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:32.415 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.412+0000 7fc81cb9f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:32.415 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.412+0000 7fc81cb9f640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:35:32.415 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.412+0000 7fc81cb9f640 10 monclient: dump: 2024-03-25T18:35:32.415 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:35:32.415 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:35:32.415 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:32.415 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:32.415 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:32.415 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:35:32.415 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:35:32.415 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:35:32.415 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:35:32.415 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:35:32.415 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.412+0000 7fc81fe05640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:32.422 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.421+0000 7fc81cb9f640 1 -- 172.21.15.155:0/3637729889 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc810001b70 con 0x7fc8181430c0 2024-03-25T18:35:32.422 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.421+0000 7fc81cb9f640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:32.422 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.421+0000 7fc81cb9f640 1 -- 172.21.15.155:0/3637729889 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fc810001f80 con 0x7fc8181430c0 2024-03-25T18:35:32.422 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.421+0000 7fc81cb9f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:32.422 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.421+0000 7fc81cb9f640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:35:32.423 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.421+0000 7fc81cb9f640 10 monclient: dump: 2024-03-25T18:35:32.423 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:35:32.423 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:35:32.423 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:32.423 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:32.423 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:32.423 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:35:32.423 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:35:32.423 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:35:32.423 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:35:32.423 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:35:32.423 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.421+0000 7fc81fe05640 10 monclient: get_monmap_and_config success 2024-03-25T18:35:32.423 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.421+0000 7fc81fe05640 4 set_mon_vals no callback set 2024-03-25T18:35:32.425 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.422+0000 7fc81fe05640 10 monclient: shutdown 2024-03-25T18:35:32.425 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.422+0000 7fc81fe05640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:32.425 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.422+0000 7fc81fe05640 1 -- 172.21.15.155:0/3637729889 >> v1:172.21.15.155:6789/0 conn(0x7fc8181430c0 legacy=0x7fc81814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:32.425 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.422+0000 7fc81fe05640 1 -- 172.21.15.155:0/3637729889 shutdown_connections 2024-03-25T18:35:32.425 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.422+0000 7fc81fe05640 1 -- 172.21.15.155:0/3637729889 >> 172.21.15.155:0/3637729889 conn(0x7fc8180b7ea0 msgr2=0x7fc8180b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:32.426 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.422+0000 7fc81fe05640 1 -- 172.21.15.155:0/3637729889 shutdown_connections 2024-03-25T18:35:32.426 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.422+0000 7fc81fe05640 1 -- 172.21.15.155:0/3637729889 wait complete. 2024-03-25T18:35:32.426 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.423+0000 7fc81fe05640 10 monclient: build_initial_monmap 2024-03-25T18:35:32.426 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.423+0000 7fc81fe05640 10 monclient: monmap: 2024-03-25T18:35:32.426 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:35:32.426 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:35:32.426 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:35:32.426 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:35:32.426 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:32.426 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:35:32.426 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:35:32.426 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:35:32.426 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:35:32.426 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:35:32.426 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.423+0000 7fc81fe05640 1 librados: starting msgr at 2024-03-25T18:35:32.426 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.423+0000 7fc81fe05640 1 librados: starting objecter 2024-03-25T18:35:32.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.423+0000 7fc81fe05640 1 Processor -- start 2024-03-25T18:35:32.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.423+0000 7fc81fe05640 1 -- start start 2024-03-25T18:35:32.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.423+0000 7fc81fe05640 1 librados: setting wanted keys 2024-03-25T18:35:32.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.423+0000 7fc81fe05640 1 librados: calling monclient init 2024-03-25T18:35:32.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.423+0000 7fc81fe05640 10 monclient: init 2024-03-25T18:35:32.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.424+0000 7fc81fe05640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:32.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.424+0000 7fc81fe05640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:35:32.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.424+0000 7fc81fe05640 10 monclient(hunting): picked mon.noname-a con 0x7fc8180a98f0 addr v1:172.21.15.155:6789/0 2024-03-25T18:35:32.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.424+0000 7fc81fe05640 10 monclient(hunting): picked mon.noname-b con 0x7fc81813f710 addr v1:172.21.15.167:6789/0 2024-03-25T18:35:32.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.424+0000 7fc81fe05640 10 monclient(hunting): picked mon.noname-c con 0x7fc8181430c0 addr v1:172.21.15.155:6790/0 2024-03-25T18:35:32.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.424+0000 7fc81fe05640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc818136e20 con 0x7fc8180a98f0 2024-03-25T18:35:32.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.424+0000 7fc81fe05640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8180b7940 con 0x7fc81813f710 2024-03-25T18:35:32.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.424+0000 7fc81fe05640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8181351a0 con 0x7fc8181430c0 2024-03-25T18:35:32.428 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.424+0000 7fc81fe05640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:32.428 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.424+0000 7fc81fe05640 10 monclient(hunting): authenticate will time out at 8178.734140s 2024-03-25T18:35:32.428 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.428+0000 7fc81d3a0640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7fc81813f710 0x7fc8181db1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:56884/0 (socket says 172.21.15.155:56884) 2024-03-25T18:35:32.428 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.428+0000 7fc81d3a0640 1 -- 172.21.15.155:0/1747893140 learned_addr learned my addr 172.21.15.155:0/1747893140 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:35:32.429 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.428+0000 7fc80e7fc640 10 client.?.objecter ms_handle_connect 0x7fc81813f710 2024-03-25T18:35:32.429 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.428+0000 7fc80e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:32.430 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.430+0000 7fc80e7fc640 10 client.?.objecter ms_handle_connect 0x7fc8181430c0 2024-03-25T18:35:32.430 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.430+0000 7fc80e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:32.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.432+0000 7fc80e7fc640 10 client.?.objecter ms_handle_connect 0x7fc8180a98f0 2024-03-25T18:35:32.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.432+0000 7fc80e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:32.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.438+0000 7fc80e7fc640 1 -- 172.21.15.155:0/1747893140 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2740644885 0 0) 0x7fc818136e20 con 0x7fc8180a98f0 2024-03-25T18:35:32.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.438+0000 7fc80e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:35:32.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.438+0000 7fc80e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:32.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.438+0000 7fc80e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:32.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.438+0000 7fc80e7fc640 10 monclient(hunting): my global_id is 4713 2024-03-25T18:35:32.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.438+0000 7fc80e7fc640 1 -- 172.21.15.155:0/1747893140 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7fc004eb0 con 0x7fc8180a98f0 2024-03-25T18:35:32.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.438+0000 7fc80e7fc640 1 -- 172.21.15.155:0/1747893140 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 141867039 0 0) 0x7fc8181351a0 con 0x7fc8181430c0 2024-03-25T18:35:32.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.438+0000 7fc80e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:35:32.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.438+0000 7fc80e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:32.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.438+0000 7fc80e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:32.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.438+0000 7fc80e7fc640 10 monclient(hunting): my global_id is 4724 2024-03-25T18:35:32.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.438+0000 7fc80e7fc640 1 -- 172.21.15.155:0/1747893140 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc818136e20 con 0x7fc8181430c0 2024-03-25T18:35:32.443 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.441+0000 7fc80e7fc640 1 -- 172.21.15.155:0/1747893140 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2506096190 0 0) 0x7fc8180b7940 con 0x7fc81813f710 2024-03-25T18:35:32.443 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.441+0000 7fc80e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:35:32.443 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.441+0000 7fc80e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:32.443 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.441+0000 7fc80e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:32.443 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.441+0000 7fc80e7fc640 10 monclient(hunting): my global_id is 4711 2024-03-25T18:35:32.443 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.441+0000 7fc80e7fc640 1 -- 172.21.15.155:0/1747893140 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8181351a0 con 0x7fc81813f710 2024-03-25T18:35:32.448 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.447+0000 7fc80e7fc640 1 -- 172.21.15.155:0/1747893140 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3906001539 0 0) 0x7fc818136e20 con 0x7fc8181430c0 2024-03-25T18:35:32.449 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.447+0000 7fc80e7fc640 1 -- 172.21.15.155:0/1747893140 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc8180b7940 con 0x7fc8181430c0 2024-03-25T18:35:32.449 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.447+0000 7fc80e7fc640 1 -- 172.21.15.155:0/1747893140 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fc810001e80 con 0x7fc8181430c0 2024-03-25T18:35:32.449 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.447+0000 7fc80e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:32.449 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.447+0000 7fc80e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:35:32.449 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.447+0000 7fc80e7fc640 10 monclient(hunting): dump: 2024-03-25T18:35:32.449 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:35:32.449 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:35:32.449 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:32.449 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:32.449 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:32.449 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:35:32.449 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:35:32.449 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:35:32.449 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:35:32.449 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:35:32.450 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.447+0000 7fc80e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:35:32.450 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.448+0000 7fc80e7fc640 1 -- 172.21.15.155:0/1747893140 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1374669349 0 0) 0x7fc7fc004eb0 con 0x7fc8180a98f0 2024-03-25T18:35:32.450 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.448+0000 7fc80e7fc640 1 -- 172.21.15.155:0/1747893140 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc818136e20 con 0x7fc8180a98f0 2024-03-25T18:35:32.450 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.449+0000 7fc80e7fc640 1 -- 172.21.15.155:0/1747893140 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fc814001930 con 0x7fc8180a98f0 2024-03-25T18:35:32.450 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.449+0000 7fc80e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:32.451 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.449+0000 7fc80e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:35:32.451 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.449+0000 7fc80e7fc640 10 monclient(hunting): dump: 2024-03-25T18:35:32.451 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:35:32.451 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:35:32.451 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:32.451 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:32.451 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:32.451 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:35:32.451 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:35:32.451 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:35:32.451 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:35:32.451 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:35:32.467 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.465+0000 7fc80e7fc640 1 -- 172.21.15.155:0/1747893140 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 165389060 0 0) 0x7fc8180b7940 con 0x7fc8181430c0 2024-03-25T18:35:32.467 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.465+0000 7fc80e7fc640 1 -- 172.21.15.155:0/1747893140 >> v1:172.21.15.167:6789/0 conn(0x7fc81813f710 legacy=0x7fc8181db1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:32.467 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.465+0000 7fc80e7fc640 1 -- 172.21.15.155:0/1747893140 >> v1:172.21.15.155:6789/0 conn(0x7fc8180a98f0 legacy=0x7fc8181daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:32.467 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.465+0000 7fc80e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:35:32.467 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.465+0000 7fc80e7fc640 1 monclient: found mon.c 2024-03-25T18:35:32.467 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.465+0000 7fc80e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:32.467 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.465+0000 7fc80e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:35:32.467 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.465+0000 7fc80e7fc640 1 -- 172.21.15.155:0/1747893140 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc818135330 con 0x7fc8181430c0 2024-03-25T18:35:32.468 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.465+0000 7fc80e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:35:32.468 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.465+0000 7fc80e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:35:32.468 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.465+0000 7fc80e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:32.468 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.465+0000 7fc81fe05640 5 monclient: authenticate success, global_id 4724 2024-03-25T18:35:32.468 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.465+0000 7fc81fe05640 10 monclient: _renew_subs 2024-03-25T18:35:32.468 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.465+0000 7fc81fe05640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:35:32.468 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.465+0000 7fc81fe05640 1 -- 172.21.15.155:0/1747893140 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc818135560 con 0x7fc8181430c0 2024-03-25T18:35:32.468 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.465+0000 7fc81fe05640 10 client.4724.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:35:32.468 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.465+0000 7fc81fe05640 10 monclient: _renew_subs 2024-03-25T18:35:32.468 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.465+0000 7fc81fe05640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:35:32.468 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.465+0000 7fc81fe05640 1 -- 172.21.15.155:0/1747893140 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc81813b490 con 0x7fc8181430c0 2024-03-25T18:35:32.468 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.465+0000 7fc81fe05640 1 librados: init done 2024-03-25T18:35:32.469 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.466+0000 7fc81fe05640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:32.469 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.467+0000 7fc81fe05640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:32.469 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.467+0000 7fc81fe05640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:35:32.469 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.467+0000 7fc81fe05640 1 -- 172.21.15.155:0/1747893140 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8180b7730 con 0x7fc8181430c0 2024-03-25T18:35:32.770 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.769+0000 7fc80e7fc640 1 -- 172.21.15.155:0/1747893140 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc810001fe0 con 0x7fc8181430c0 2024-03-25T18:35:32.770 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.769+0000 7fc80e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:32.770 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.769+0000 7fc80e7fc640 1 -- 172.21.15.155:0/1747893140 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fc810004fb0 con 0x7fc8181430c0 2024-03-25T18:35:32.770 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.769+0000 7fc80e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:32.770 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.769+0000 7fc80e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:35:32.770 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.769+0000 7fc80e7fc640 10 monclient: dump: 2024-03-25T18:35:32.770 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:35:32.770 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:35:32.770 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:32.770 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:32.770 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:32.771 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:35:32.771 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:35:32.771 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:35:32.771 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:35:32.771 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:35:32.771 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.769+0000 7fc80ffff640 4 set_mon_vals no callback set 2024-03-25T18:35:32.771 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.770+0000 7fc80e7fc640 1 -- 172.21.15.155:0/1747893140 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7fc81001af80 con 0x7fc8181430c0 2024-03-25T18:35:32.772 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.770+0000 7fc81dba1640 10 monclient: get_auth_request con 0x7fc7fc06df70 auth_method 0 2024-03-25T18:35:32.772 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.770+0000 7fc80e7fc640 10 client.4724.objecter ms_handle_connect 0x7fc7fc06df70 2024-03-25T18:35:32.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.775+0000 7fc80e7fc640 1 -- 172.21.15.155:0/1747893140 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1555029372 0 0) 0x7fc810054a40 con 0x7fc8181430c0 2024-03-25T18:35:32.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.775+0000 7fc80e7fc640 10 client.4724.objecter ms_dispatch 0x7fc818000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:35:32.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.775+0000 7fc80e7fc640 3 client.4724.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:35:32.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.775+0000 7fc80e7fc640 3 client.4724.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:35:32.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.775+0000 7fc80e7fc640 20 client.4724.objecter dump_active .. 0 homeless 2024-03-25T18:35:32.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.775+0000 7fc80e7fc640 1 -- 172.21.15.155:0/1747893140 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc810050550 con 0x7fc8181430c0 2024-03-25T18:35:32.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.775+0000 7fc80e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:32.777 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:32.775+0000 7fc80e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:35:32.913 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:11:06.518 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:32.928 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:11:06.532 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:32.968 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:11:06.573 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:32.983 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:11:06.587 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:32.995 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:11:06.600 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:33.003 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:11:06.582 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:33.007 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:11:06.586 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:33.010 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:33.009+0000 7fc81fe05640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:33.010 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:33.009+0000 7fc81fe05640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:33.010 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:33.009+0000 7fc81fe05640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-25T18:35:33.010 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:33.009+0000 7fc81fe05640 1 -- 172.21.15.155:0/1747893140 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc8180627b0 con 0x7fc8181430c0 2024-03-25T18:35:33.040 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:11:06.619 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:33.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:33.047+0000 7fc80e7fc640 1 -- 172.21.15.155:0/1747893140 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3170230695) 0x7fc810052480 con 0x7fc8181430c0 2024-03-25T18:35:33.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:33.047+0000 7fc80e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:33.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:33.047+0000 7fc80e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:35:33.048 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:35:33.048 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":11,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:34:47.433406+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:34:47.433406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:35:33.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:33.049+0000 7fc81fe05640 10 librados: watch_flush enter 2024-03-25T18:35:33.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:33.049+0000 7fc81fe05640 10 librados: watch_flush exit 2024-03-25T18:35:33.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:33.049+0000 7fc81fe05640 20 client.4724.objecter shutdown clearing up homeless session... 2024-03-25T18:35:33.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:33.049+0000 7fc81fe05640 10 client.4724.objecter successfully canceled tick 2024-03-25T18:35:33.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:33.049+0000 7fc81fe05640 1 -- 172.21.15.155:0/1747893140 >> v1:172.21.15.167:6800/2919070503 conn(0x7fc7fc06df70 legacy=0x7fc7fc070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:33.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:33.049+0000 7fc81fe05640 10 monclient: shutdown 2024-03-25T18:35:33.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:33.049+0000 7fc81fe05640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:33.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:33.049+0000 7fc81fe05640 1 -- 172.21.15.155:0/1747893140 >> v1:172.21.15.155:6790/0 conn(0x7fc8181430c0 legacy=0x7fc8181e1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:33.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:33.049+0000 7fc81fe05640 1 -- 172.21.15.155:0/1747893140 shutdown_connections 2024-03-25T18:35:33.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:33.049+0000 7fc81fe05640 1 -- 172.21.15.155:0/1747893140 >> 172.21.15.155:0/1747893140 conn(0x7fc8180b7ea0 msgr2=0x7fc818074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:33.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:33.049+0000 7fc81fe05640 1 -- 172.21.15.155:0/1747893140 shutdown_connections 2024-03-25T18:35:33.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:33.049+0000 7fc81fe05640 1 -- 172.21.15.155:0/1747893140 wait complete. 2024-03-25T18:35:33.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:33.049+0000 7fc81fe05640 1 librados: shutdown 2024-03-25T18:35:33.060 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:35:33.067 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:11:06.672 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:39.868 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:35:39.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:39.984+0000 7fe803797640 10 monclient: get_monmap_and_config 2024-03-25T18:35:39.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:39.984+0000 7fe803797640 10 monclient: build_initial_monmap 2024-03-25T18:35:39.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:39.984+0000 7fe803797640 10 monclient: monmap: 2024-03-25T18:35:39.986 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:35:39.986 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:35:39.986 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:35:39.985737+0000 2024-03-25T18:35:39.986 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:35:39.985737+0000 2024-03-25T18:35:39.986 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:39.987 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:35:39.987 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:35:39.987 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:35:39.987 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:35:39.987 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:35:39.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:39.985+0000 7fe803797640 1 Processor -- start 2024-03-25T18:35:39.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:39.985+0000 7fe803797640 1 -- start start 2024-03-25T18:35:39.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:39.985+0000 7fe803797640 10 monclient: init 2024-03-25T18:35:39.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:39.986+0000 7fe803797640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:39.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:39.986+0000 7fe803797640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:35:39.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:39.986+0000 7fe803797640 10 monclient(hunting): picked mon.noname-a con 0x7fe7fc061830 addr v1:172.21.15.155:6789/0 2024-03-25T18:35:39.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:39.986+0000 7fe803797640 10 monclient(hunting): picked mon.noname-b con 0x7fe7fc1400e0 addr v1:172.21.15.155:6790/0 2024-03-25T18:35:39.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:39.986+0000 7fe803797640 10 monclient(hunting): picked mon.noname-c con 0x7fe7fc1499d0 addr v1:172.21.15.167:6789/0 2024-03-25T18:35:39.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:39.986+0000 7fe803797640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7fc072600 con 0x7fe7fc061830 2024-03-25T18:35:39.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:39.986+0000 7fe803797640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7fc140c80 con 0x7fe7fc1499d0 2024-03-25T18:35:39.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:39.986+0000 7fe803797640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7fc072830 con 0x7fe7fc1400e0 2024-03-25T18:35:39.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:39.986+0000 7fe803797640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:39.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:39.986+0000 7fe803797640 10 monclient(hunting): authenticate will time out at 8186.296360s 2024-03-25T18:35:39.997 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:39.995+0000 7fe800d32640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fe7fc1400e0 0x7fe7fc140570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:35286/0 (socket says 172.21.15.155:35286) 2024-03-25T18:35:39.997 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:39.995+0000 7fe800d32640 1 -- 172.21.15.155:0/2733208710 learned_addr learned my addr 172.21.15.155:0/2733208710 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:35:40.003 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.001+0000 7fe7f3fff640 1 -- 172.21.15.155:0/2733208710 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1581497884 0 0) 0x7fe7fc072830 con 0x7fe7fc1400e0 2024-03-25T18:35:40.003 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.001+0000 7fe7f3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:35:40.003 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.001+0000 7fe7f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:40.003 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.001+0000 7fe7f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:40.003 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.001+0000 7fe7f3fff640 10 monclient(hunting): my global_id is 4727 2024-03-25T18:35:40.003 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.001+0000 7fe7f3fff640 1 -- 172.21.15.155:0/2733208710 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7e8003520 con 0x7fe7fc1400e0 2024-03-25T18:35:40.011 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.009+0000 7fe7f3fff640 1 -- 172.21.15.155:0/2733208710 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1911497865 0 0) 0x7fe7fc140c80 con 0x7fe7fc1499d0 2024-03-25T18:35:40.011 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.009+0000 7fe7f3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:35:40.011 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.009+0000 7fe7f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:40.011 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.009+0000 7fe7f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:40.011 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.009+0000 7fe7f3fff640 10 monclient(hunting): my global_id is 4714 2024-03-25T18:35:40.011 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.009+0000 7fe7f3fff640 1 -- 172.21.15.155:0/2733208710 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7fc072830 con 0x7fe7fc1499d0 2024-03-25T18:35:40.019 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.018+0000 7fe7f3fff640 1 -- 172.21.15.155:0/2733208710 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4222142624 0 0) 0x7fe7fc072600 con 0x7fe7fc061830 2024-03-25T18:35:40.019 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.018+0000 7fe7f3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:35:40.019 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.018+0000 7fe7f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:40.019 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.018+0000 7fe7f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:40.019 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.018+0000 7fe7f3fff640 10 monclient(hunting): my global_id is 4716 2024-03-25T18:35:40.019 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.018+0000 7fe7f3fff640 1 -- 172.21.15.155:0/2733208710 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7fc140c80 con 0x7fe7fc061830 2024-03-25T18:35:40.031 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.029+0000 7fe7f3fff640 1 -- 172.21.15.155:0/2733208710 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2990092876 0 0) 0x7fe7e8003520 con 0x7fe7fc1400e0 2024-03-25T18:35:40.031 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.029+0000 7fe7f3fff640 1 -- 172.21.15.155:0/2733208710 >> v1:172.21.15.167:6789/0 conn(0x7fe7fc1499d0 legacy=0x7fe7fc14bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:40.031 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.029+0000 7fe7f3fff640 1 -- 172.21.15.155:0/2733208710 >> v1:172.21.15.155:6789/0 conn(0x7fe7fc061830 legacy=0x7fe7fc13d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:40.031 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.029+0000 7fe7f3fff640 10 monclient: _finish_hunting 0 2024-03-25T18:35:40.031 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.029+0000 7fe7f3fff640 1 monclient: found mon.noname-b 2024-03-25T18:35:40.031 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.029+0000 7fe7f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:40.031 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.029+0000 7fe7f3fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-25T18:35:40.031 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.029+0000 7fe7f3fff640 1 -- 172.21.15.155:0/2733208710 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7fc072b10 con 0x7fe7fc1400e0 2024-03-25T18:35:40.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.029+0000 7fe7f3fff640 10 monclient: _finish_auth 0 2024-03-25T18:35:40.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.029+0000 7fe7f3fff640 10 monclient: _check_auth_tickets 2024-03-25T18:35:40.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.029+0000 7fe7f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:40.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.029+0000 7fe7f3fff640 1 -- 172.21.15.155:0/2733208710 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fe7ec001690 con 0x7fe7fc1400e0 2024-03-25T18:35:40.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.029+0000 7fe803797640 5 monclient: authenticate success, global_id 4727 2024-03-25T18:35:40.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.029+0000 7fe803797640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:40.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.029+0000 7fe7f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:40.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.029+0000 7fe7f3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:35:40.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.029+0000 7fe7f3fff640 10 monclient: dump: 2024-03-25T18:35:40.032 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:35:40.033 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:35:40.033 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:40.033 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:40.033 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:40.033 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:35:40.033 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:35:40.033 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:35:40.033 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:35:40.033 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:35:40.033 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.029+0000 7fe803797640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:40.042 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.041+0000 7fe7f3fff640 1 -- 172.21.15.155:0/2733208710 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe7ec0017f0 con 0x7fe7fc1400e0 2024-03-25T18:35:40.042 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.041+0000 7fe7f3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:40.042 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.041+0000 7fe7f3fff640 1 -- 172.21.15.155:0/2733208710 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fe7ec001c00 con 0x7fe7fc1400e0 2024-03-25T18:35:40.043 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.041+0000 7fe7f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:40.043 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.041+0000 7fe7f3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:35:40.043 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.041+0000 7fe7f3fff640 10 monclient: dump: 2024-03-25T18:35:40.043 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:35:40.043 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:35:40.043 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:40.043 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:40.043 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:40.043 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:35:40.043 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:35:40.044 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:35:40.044 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:35:40.044 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:35:40.044 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.041+0000 7fe803797640 10 monclient: get_monmap_and_config success 2024-03-25T18:35:40.044 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.041+0000 7fe803797640 4 set_mon_vals no callback set 2024-03-25T18:35:40.047 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.042+0000 7fe803797640 10 monclient: shutdown 2024-03-25T18:35:40.047 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.042+0000 7fe803797640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:40.047 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.042+0000 7fe803797640 1 -- 172.21.15.155:0/2733208710 >> v1:172.21.15.155:6790/0 conn(0x7fe7fc1400e0 legacy=0x7fe7fc140570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:40.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.042+0000 7fe803797640 1 -- 172.21.15.155:0/2733208710 shutdown_connections 2024-03-25T18:35:40.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.042+0000 7fe803797640 1 -- 172.21.15.155:0/2733208710 >> 172.21.15.155:0/2733208710 conn(0x7fe7fc0b0ef0 msgr2=0x7fe7fc0b1300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:40.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.042+0000 7fe803797640 1 -- 172.21.15.155:0/2733208710 shutdown_connections 2024-03-25T18:35:40.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.042+0000 7fe803797640 1 -- 172.21.15.155:0/2733208710 wait complete. 2024-03-25T18:35:40.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.043+0000 7fe803797640 10 monclient: build_initial_monmap 2024-03-25T18:35:40.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.043+0000 7fe803797640 10 monclient: monmap: 2024-03-25T18:35:40.048 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:35:40.048 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:35:40.049 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:35:40.049 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:35:40.049 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:40.049 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:35:40.049 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:35:40.049 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:35:40.049 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:35:40.049 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:35:40.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.043+0000 7fe803797640 1 librados: starting msgr at 2024-03-25T18:35:40.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.043+0000 7fe803797640 1 librados: starting objecter 2024-03-25T18:35:40.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.043+0000 7fe803797640 1 Processor -- start 2024-03-25T18:35:40.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.043+0000 7fe803797640 1 -- start start 2024-03-25T18:35:40.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.043+0000 7fe803797640 1 librados: setting wanted keys 2024-03-25T18:35:40.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.043+0000 7fe803797640 1 librados: calling monclient init 2024-03-25T18:35:40.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.043+0000 7fe803797640 10 monclient: init 2024-03-25T18:35:40.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.044+0000 7fe803797640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:40.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.044+0000 7fe803797640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:35:40.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.044+0000 7fe803797640 10 monclient(hunting): picked mon.noname-c con 0x7fe7fc061830 addr v1:172.21.15.155:6790/0 2024-03-25T18:35:40.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.044+0000 7fe803797640 10 monclient(hunting): picked mon.noname-b con 0x7fe7fc1400e0 addr v1:172.21.15.167:6789/0 2024-03-25T18:35:40.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.044+0000 7fe803797640 10 monclient(hunting): picked mon.noname-a con 0x7fe7fc1499d0 addr v1:172.21.15.155:6789/0 2024-03-25T18:35:40.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.044+0000 7fe803797640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7fc1da9c0 con 0x7fe7fc1499d0 2024-03-25T18:35:40.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.044+0000 7fe803797640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7fc1372d0 con 0x7fe7fc1400e0 2024-03-25T18:35:40.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.044+0000 7fe803797640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7fc135b90 con 0x7fe7fc061830 2024-03-25T18:35:40.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.044+0000 7fe803797640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:40.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.044+0000 7fe803797640 10 monclient(hunting): authenticate will time out at 8186.354072s 2024-03-25T18:35:40.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.046+0000 7fe800d32640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7fe7fc1400e0 0x7fe7fc1daea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:37140/0 (socket says 172.21.15.155:37140) 2024-03-25T18:35:40.051 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.046+0000 7fe800d32640 1 -- 172.21.15.155:0/3032653224 learned_addr learned my addr 172.21.15.155:0/3032653224 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:35:40.052 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.047+0000 7fe7f1ffb640 10 client.?.objecter ms_handle_connect 0x7fe7fc1400e0 2024-03-25T18:35:40.053 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.047+0000 7fe7f1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:40.053 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.050+0000 7fe7f1ffb640 10 client.?.objecter ms_handle_connect 0x7fe7fc061830 2024-03-25T18:35:40.053 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.050+0000 7fe7f1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:40.053 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.052+0000 7fe7f1ffb640 1 -- 172.21.15.155:0/3032653224 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2693518320 0 0) 0x7fe7fc1372d0 con 0x7fe7fc1400e0 2024-03-25T18:35:40.054 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.052+0000 7fe7f1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:35:40.054 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.052+0000 7fe7f1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:40.054 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.052+0000 7fe7f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:40.054 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.052+0000 7fe7f1ffb640 10 monclient(hunting): my global_id is 4717 2024-03-25T18:35:40.054 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.052+0000 7fe7f1ffb640 1 -- 172.21.15.155:0/3032653224 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7e0004eb0 con 0x7fe7fc1400e0 2024-03-25T18:35:40.060 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.059+0000 7fe7f1ffb640 1 -- 172.21.15.155:0/3032653224 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1981441753 0 0) 0x7fe7fc135b90 con 0x7fe7fc061830 2024-03-25T18:35:40.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.059+0000 7fe7f1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:35:40.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.059+0000 7fe7f1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:40.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.059+0000 7fe7f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:40.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.059+0000 7fe7f1ffb640 10 monclient(hunting): my global_id is 4730 2024-03-25T18:35:40.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.059+0000 7fe7f1ffb640 1 -- 172.21.15.155:0/3032653224 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7fc1372d0 con 0x7fe7fc061830 2024-03-25T18:35:40.062 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.060+0000 7fe7f1ffb640 10 client.?.objecter ms_handle_connect 0x7fe7fc1499d0 2024-03-25T18:35:40.062 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.060+0000 7fe7f1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:40.067 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.066+0000 7fe7f1ffb640 1 -- 172.21.15.155:0/3032653224 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 563932654 0 0) 0x7fe7e0004eb0 con 0x7fe7fc1400e0 2024-03-25T18:35:40.067 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.066+0000 7fe7f1ffb640 1 -- 172.21.15.155:0/3032653224 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7fc135b90 con 0x7fe7fc1400e0 2024-03-25T18:35:40.068 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.066+0000 7fe7f1ffb640 1 -- 172.21.15.155:0/3032653224 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fe7ec001bc0 con 0x7fe7fc1400e0 2024-03-25T18:35:40.068 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.066+0000 7fe7f1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:40.068 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.066+0000 7fe7f1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:35:40.068 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.066+0000 7fe7f1ffb640 10 monclient(hunting): dump: 2024-03-25T18:35:40.068 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:35:40.068 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:35:40.068 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:40.068 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:40.068 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:40.068 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:35:40.069 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:35:40.069 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:35:40.069 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:35:40.069 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:35:40.069 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.066+0000 7fe7f1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:35:40.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.071+0000 7fe7f1ffb640 1 -- 172.21.15.155:0/3032653224 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2213529930 0 0) 0x7fe7fc1da9c0 con 0x7fe7fc1499d0 2024-03-25T18:35:40.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.071+0000 7fe7f1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:35:40.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.071+0000 7fe7f1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:40.073 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.071+0000 7fe7f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:40.073 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.071+0000 7fe7f1ffb640 10 monclient(hunting): my global_id is 4719 2024-03-25T18:35:40.073 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.071+0000 7fe7f1ffb640 1 -- 172.21.15.155:0/3032653224 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7e0004eb0 con 0x7fe7fc1499d0 2024-03-25T18:35:40.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.074+0000 7fe7f1ffb640 1 -- 172.21.15.155:0/3032653224 <== mon.1 v1:172.21.15.167:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4054978793 0 0) 0x7fe7fc135b90 con 0x7fe7fc1400e0 2024-03-25T18:35:40.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.074+0000 7fe7f1ffb640 1 -- 172.21.15.155:0/3032653224 >> v1:172.21.15.155:6790/0 conn(0x7fe7fc061830 legacy=0x7fe7fc135480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:40.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.075+0000 7fe7f1ffb640 1 -- 172.21.15.155:0/3032653224 >> v1:172.21.15.155:6789/0 conn(0x7fe7fc1499d0 legacy=0x7fe7fc133500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:40.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.075+0000 7fe7f1ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:35:40.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.075+0000 7fe7f1ffb640 1 monclient: found mon.b 2024-03-25T18:35:40.078 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.075+0000 7fe7f1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:40.078 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.075+0000 7fe7f1ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:35:40.078 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.075+0000 7fe7f1ffb640 1 -- 172.21.15.155:0/3032653224 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7fc137e60 con 0x7fe7fc1400e0 2024-03-25T18:35:40.078 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.075+0000 7fe7f1ffb640 10 monclient: _finish_auth 0 2024-03-25T18:35:40.078 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.075+0000 7fe7f1ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:35:40.078 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.075+0000 7fe7f1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:40.078 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.075+0000 7fe803797640 5 monclient: authenticate success, global_id 4717 2024-03-25T18:35:40.078 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.075+0000 7fe803797640 10 monclient: _renew_subs 2024-03-25T18:35:40.079 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.075+0000 7fe803797640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:35:40.079 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.075+0000 7fe803797640 1 -- 172.21.15.155:0/3032653224 --> v1:172.21.15.167:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7fc1db5b0 con 0x7fe7fc1400e0 2024-03-25T18:35:40.079 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.075+0000 7fe803797640 10 client.4717.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:35:40.079 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.075+0000 7fe803797640 10 monclient: _renew_subs 2024-03-25T18:35:40.079 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.075+0000 7fe803797640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:35:40.079 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.075+0000 7fe803797640 1 -- 172.21.15.155:0/3032653224 --> v1:172.21.15.167:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7fc133df0 con 0x7fe7fc1400e0 2024-03-25T18:35:40.079 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.075+0000 7fe803797640 1 librados: init done 2024-03-25T18:35:40.079 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.076+0000 7fe803797640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:40.079 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.076+0000 7fe803797640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:40.079 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.076+0000 7fe803797640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:35:40.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.076+0000 7fe803797640 1 -- 172.21.15.155:0/3032653224 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7fc0627b0 con 0x7fe7fc1400e0 2024-03-25T18:35:40.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.359+0000 7fe7f1ffb640 1 -- 172.21.15.155:0/3032653224 <== mon.1 v1:172.21.15.167:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe7ec0011f0 con 0x7fe7fc1400e0 2024-03-25T18:35:40.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.359+0000 7fe7f1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:40.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.359+0000 7fe7f1ffb640 1 -- 172.21.15.155:0/3032653224 <== mon.1 v1:172.21.15.167:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7fe7ec004cb0 con 0x7fe7fc1400e0 2024-03-25T18:35:40.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.359+0000 7fe7f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:40.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.359+0000 7fe7f1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:35:40.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.359+0000 7fe7f1ffb640 10 monclient: dump: 2024-03-25T18:35:40.361 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:35:40.362 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:35:40.362 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:40.362 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:40.362 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:40.362 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:35:40.362 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:35:40.362 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:35:40.362 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:35:40.362 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:35:40.362 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.359+0000 7fe7f37fe640 4 set_mon_vals no callback set 2024-03-25T18:35:40.362 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.360+0000 7fe7f1ffb640 1 -- 172.21.15.155:0/3032653224 <== mon.1 v1:172.21.15.167:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7fe7ec01ad00 con 0x7fe7fc1400e0 2024-03-25T18:35:40.363 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.360+0000 7fe801533640 10 monclient: get_auth_request con 0x7fe7e006ddb0 auth_method 0 2024-03-25T18:35:40.363 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.361+0000 7fe7f1ffb640 10 client.4717.objecter ms_handle_connect 0x7fe7e006ddb0 2024-03-25T18:35:40.367 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.365+0000 7fe7f1ffb640 1 -- 172.21.15.155:0/3032653224 <== mon.1 v1:172.21.15.167:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1555029372 0 0) 0x7fe7ec054a00 con 0x7fe7fc1400e0 2024-03-25T18:35:40.367 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.365+0000 7fe7f1ffb640 10 client.4717.objecter ms_dispatch 0x7fe7fc000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:35:40.367 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.365+0000 7fe7f1ffb640 3 client.4717.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:35:40.367 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.365+0000 7fe7f1ffb640 3 client.4717.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:35:40.367 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.365+0000 7fe7f1ffb640 20 client.4717.objecter dump_active .. 0 homeless 2024-03-25T18:35:40.367 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.366+0000 7fe7f1ffb640 1 -- 172.21.15.155:0/3032653224 <== mon.1 v1:172.21.15.167:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe7ec050340 con 0x7fe7fc1400e0 2024-03-25T18:35:40.367 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.366+0000 7fe7f1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:40.368 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.366+0000 7fe7f1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:35:40.623 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.622+0000 7fe803797640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:40.623 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.622+0000 7fe803797640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:40.624 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.622+0000 7fe803797640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:35:40.624 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.622+0000 7fe803797640 1 -- 172.21.15.155:0/3032653224 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe7fc13f6f0 con 0x7fe7fc1400e0 2024-03-25T18:35:40.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.649+0000 7fe7f1ffb640 1 -- 172.21.15.155:0/3032653224 <== mon.1 v1:172.21.15.167:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3170230695) 0x7fe7ec052440 con 0x7fe7fc1400e0 2024-03-25T18:35:40.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.649+0000 7fe7f1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:40.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.649+0000 7fe7f1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:35:40.651 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:35:40.651 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":11,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:34:47.433406+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:34:47.433406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:35:40.653 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.652+0000 7fe803797640 10 librados: watch_flush enter 2024-03-25T18:35:40.654 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.652+0000 7fe803797640 10 librados: watch_flush exit 2024-03-25T18:35:40.654 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.652+0000 7fe803797640 20 client.4717.objecter shutdown clearing up homeless session... 2024-03-25T18:35:40.654 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.652+0000 7fe803797640 10 client.4717.objecter successfully canceled tick 2024-03-25T18:35:40.654 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.652+0000 7fe803797640 1 -- 172.21.15.155:0/3032653224 >> v1:172.21.15.167:6800/2919070503 conn(0x7fe7e006ddb0 legacy=0x7fe7e0070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:40.654 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.652+0000 7fe803797640 10 monclient: shutdown 2024-03-25T18:35:40.654 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.652+0000 7fe803797640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:40.654 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.652+0000 7fe803797640 1 -- 172.21.15.155:0/3032653224 >> v1:172.21.15.167:6789/0 conn(0x7fe7fc1400e0 legacy=0x7fe7fc1daea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:40.654 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.652+0000 7fe803797640 1 -- 172.21.15.155:0/3032653224 shutdown_connections 2024-03-25T18:35:40.654 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.652+0000 7fe803797640 1 -- 172.21.15.155:0/3032653224 >> 172.21.15.155:0/3032653224 conn(0x7fe7fc0b0ef0 msgr2=0x7fe7fc140e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:40.654 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.652+0000 7fe803797640 1 -- 172.21.15.155:0/3032653224 shutdown_connections 2024-03-25T18:35:40.654 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.652+0000 7fe803797640 1 -- 172.21.15.155:0/3032653224 wait complete. 2024-03-25T18:35:40.654 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:40.652+0000 7fe803797640 1 librados: shutdown 2024-03-25T18:35:40.666 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:35:47.473 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:35:47.590 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.589+0000 7f78e331c640 10 monclient: get_monmap_and_config 2024-03-25T18:35:47.590 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.589+0000 7f78e331c640 10 monclient: build_initial_monmap 2024-03-25T18:35:47.590 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.589+0000 7f78e331c640 10 monclient: monmap: 2024-03-25T18:35:47.590 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:35:47.590 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:35:47.590 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:35:47.590259+0000 2024-03-25T18:35:47.590 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:35:47.590259+0000 2024-03-25T18:35:47.590 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:47.591 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:35:47.591 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:35:47.591 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:35:47.591 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:35:47.591 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:35:47.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.589+0000 7f78e331c640 1 Processor -- start 2024-03-25T18:35:47.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.590+0000 7f78e331c640 1 -- start start 2024-03-25T18:35:47.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.590+0000 7f78e331c640 10 monclient: init 2024-03-25T18:35:47.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.591+0000 7f78e331c640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:47.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.591+0000 7f78e331c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:35:47.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.591+0000 7f78e331c640 10 monclient(hunting): picked mon.noname-b con 0x7f78dc0a98f0 addr v1:172.21.15.155:6790/0 2024-03-25T18:35:47.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.591+0000 7f78e331c640 10 monclient(hunting): picked mon.noname-a con 0x7f78dc13f710 addr v1:172.21.15.155:6789/0 2024-03-25T18:35:47.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.591+0000 7f78e331c640 10 monclient(hunting): picked mon.noname-c con 0x7f78dc1430c0 addr v1:172.21.15.167:6789/0 2024-03-25T18:35:47.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.591+0000 7f78e331c640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78dc142520 con 0x7f78dc13f710 2024-03-25T18:35:47.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.591+0000 7f78e331c640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78dc0b28a0 con 0x7f78dc1430c0 2024-03-25T18:35:47.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.591+0000 7f78e331c640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78dc0b2a30 con 0x7f78dc0a98f0 2024-03-25T18:35:47.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.591+0000 7f78e331c640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:47.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.591+0000 7f78e331c640 10 monclient(hunting): authenticate will time out at 8193.901110s 2024-03-25T18:35:47.606 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.605+0000 7f78e08b7640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f78dc13f710 0x7f78dc1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:60922/0 (socket says 172.21.15.155:60922) 2024-03-25T18:35:47.606 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.605+0000 7f78e08b7640 1 -- 172.21.15.155:0/1576043005 learned_addr learned my addr 172.21.15.155:0/1576043005 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:35:47.614 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.613+0000 7f78d3fff640 1 -- 172.21.15.155:0/1576043005 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 604485560 0 0) 0x7f78dc142520 con 0x7f78dc13f710 2024-03-25T18:35:47.614 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.613+0000 7f78d3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:35:47.614 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.613+0000 7f78d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:47.614 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.613+0000 7f78d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:47.614 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.613+0000 7f78d3fff640 10 monclient(hunting): my global_id is 4722 2024-03-25T18:35:47.614 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.613+0000 7f78d3fff640 1 -- 172.21.15.155:0/1576043005 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78c8003520 con 0x7f78dc13f710 2024-03-25T18:35:47.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.621+0000 7f78d3fff640 1 -- 172.21.15.155:0/1576043005 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2871372317 0 0) 0x7f78dc0b28a0 con 0x7f78dc1430c0 2024-03-25T18:35:47.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.621+0000 7f78d3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:35:47.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.621+0000 7f78d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:47.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.621+0000 7f78d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:47.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.621+0000 7f78d3fff640 10 monclient(hunting): my global_id is 4720 2024-03-25T18:35:47.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.621+0000 7f78d3fff640 1 -- 172.21.15.155:0/1576043005 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78dc142520 con 0x7f78dc1430c0 2024-03-25T18:35:47.623 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.622+0000 7f78d3fff640 1 -- 172.21.15.155:0/1576043005 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4071698568 0 0) 0x7f78dc0b2a30 con 0x7f78dc0a98f0 2024-03-25T18:35:47.623 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.622+0000 7f78d3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:35:47.623 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.622+0000 7f78d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:47.623 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.622+0000 7f78d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:47.624 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.622+0000 7f78d3fff640 10 monclient(hunting): my global_id is 4733 2024-03-25T18:35:47.624 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.622+0000 7f78d3fff640 1 -- 172.21.15.155:0/1576043005 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78dc0b28a0 con 0x7f78dc0a98f0 2024-03-25T18:35:47.626 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.624+0000 7f78d3fff640 1 -- 172.21.15.155:0/1576043005 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3867273250 0 0) 0x7f78c8003520 con 0x7f78dc13f710 2024-03-25T18:35:47.626 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.624+0000 7f78d3fff640 1 -- 172.21.15.155:0/1576043005 >> v1:172.21.15.155:6790/0 conn(0x7f78dc0a98f0 legacy=0x7f78dc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:47.626 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.624+0000 7f78d3fff640 1 -- 172.21.15.155:0/1576043005 >> v1:172.21.15.167:6789/0 conn(0x7f78dc1430c0 legacy=0x7f78dc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:47.626 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.624+0000 7f78d3fff640 10 monclient: _finish_hunting 0 2024-03-25T18:35:47.626 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.624+0000 7f78d3fff640 1 monclient: found mon.noname-a 2024-03-25T18:35:47.626 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.624+0000 7f78d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:47.626 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.624+0000 7f78d3fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-25T18:35:47.626 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.624+0000 7f78d3fff640 1 -- 172.21.15.155:0/1576043005 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78dc0b44c0 con 0x7f78dc13f710 2024-03-25T18:35:47.626 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.624+0000 7f78d3fff640 10 monclient: _finish_auth 0 2024-03-25T18:35:47.626 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.624+0000 7f78d3fff640 10 monclient: _check_auth_tickets 2024-03-25T18:35:47.626 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.624+0000 7f78d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:47.626 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.624+0000 7f78d3fff640 1 -- 172.21.15.155:0/1576043005 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f78cc001690 con 0x7f78dc13f710 2024-03-25T18:35:47.627 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.624+0000 7f78d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:47.627 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.624+0000 7f78d3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:35:47.627 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.624+0000 7f78d3fff640 10 monclient: dump: 2024-03-25T18:35:47.627 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:35:47.627 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:35:47.627 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:47.627 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:47.627 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:47.627 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:35:47.627 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:35:47.627 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:35:47.627 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:35:47.627 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:35:47.627 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.625+0000 7f78e331c640 5 monclient: authenticate success, global_id 4722 2024-03-25T18:35:47.628 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.625+0000 7f78e331c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:47.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.638+0000 7f78d3fff640 1 -- 172.21.15.155:0/1576043005 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f78cc0017f0 con 0x7f78dc13f710 2024-03-25T18:35:47.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.638+0000 7f78d3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:47.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.638+0000 7f78d3fff640 1 -- 172.21.15.155:0/1576043005 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f78cc001c00 con 0x7f78dc13f710 2024-03-25T18:35:47.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.638+0000 7f78e331c640 10 monclient: get_monmap_and_config success 2024-03-25T18:35:47.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.638+0000 7f78e331c640 4 set_mon_vals no callback set 2024-03-25T18:35:47.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.638+0000 7f78d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:47.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.638+0000 7f78d3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:35:47.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.638+0000 7f78d3fff640 10 monclient: dump: 2024-03-25T18:35:47.640 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:35:47.640 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:35:47.640 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:47.640 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:47.640 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:47.640 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:35:47.640 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:35:47.640 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:35:47.640 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:35:47.640 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:35:47.642 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.639+0000 7f78e331c640 10 monclient: shutdown 2024-03-25T18:35:47.642 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.639+0000 7f78e331c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:47.642 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.639+0000 7f78e331c640 1 -- 172.21.15.155:0/1576043005 >> v1:172.21.15.155:6789/0 conn(0x7f78dc13f710 legacy=0x7f78dc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:47.642 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.639+0000 7f78e331c640 1 -- 172.21.15.155:0/1576043005 shutdown_connections 2024-03-25T18:35:47.642 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.639+0000 7f78e331c640 1 -- 172.21.15.155:0/1576043005 >> 172.21.15.155:0/1576043005 conn(0x7f78dc0b7ea0 msgr2=0x7f78dc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:47.642 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.640+0000 7f78e331c640 1 -- 172.21.15.155:0/1576043005 shutdown_connections 2024-03-25T18:35:47.642 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.640+0000 7f78e331c640 1 -- 172.21.15.155:0/1576043005 wait complete. 2024-03-25T18:35:47.642 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.640+0000 7f78e331c640 10 monclient: build_initial_monmap 2024-03-25T18:35:47.642 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.640+0000 7f78e331c640 10 monclient: monmap: 2024-03-25T18:35:47.643 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:35:47.643 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:35:47.643 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:35:47.643 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:35:47.643 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:47.643 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:35:47.643 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:35:47.643 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:35:47.643 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:35:47.643 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:35:47.643 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.641+0000 7f78e331c640 1 librados: starting msgr at 2024-03-25T18:35:47.643 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.641+0000 7f78e331c640 1 librados: starting objecter 2024-03-25T18:35:47.643 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.641+0000 7f78e331c640 1 Processor -- start 2024-03-25T18:35:47.643 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.641+0000 7f78e331c640 1 -- start start 2024-03-25T18:35:47.643 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.641+0000 7f78e331c640 1 librados: setting wanted keys 2024-03-25T18:35:47.643 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.641+0000 7f78e331c640 1 librados: calling monclient init 2024-03-25T18:35:47.644 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.641+0000 7f78e331c640 10 monclient: init 2024-03-25T18:35:47.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.642+0000 7f78e331c640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:47.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.642+0000 7f78e331c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:35:47.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.642+0000 7f78e331c640 10 monclient(hunting): picked mon.noname-c con 0x7f78dc0a98f0 addr v1:172.21.15.155:6790/0 2024-03-25T18:35:47.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.642+0000 7f78e331c640 10 monclient(hunting): picked mon.noname-b con 0x7f78dc13f710 addr v1:172.21.15.167:6789/0 2024-03-25T18:35:47.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.642+0000 7f78e331c640 10 monclient(hunting): picked mon.noname-a con 0x7f78dc1430c0 addr v1:172.21.15.155:6789/0 2024-03-25T18:35:47.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.642+0000 7f78e331c640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78dc1db930 con 0x7f78dc1430c0 2024-03-25T18:35:47.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.642+0000 7f78e331c640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78dc070210 con 0x7f78dc13f710 2024-03-25T18:35:47.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.642+0000 7f78e331c640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78dc070480 con 0x7f78dc0a98f0 2024-03-25T18:35:47.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.642+0000 7f78e331c640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:47.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.642+0000 7f78e331c640 10 monclient(hunting): authenticate will time out at 8193.952305s 2024-03-25T18:35:47.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.649+0000 7f78e18b9640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f78dc1430c0 0x7f78dc1e6280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:60962/0 (socket says 172.21.15.155:60962) 2024-03-25T18:35:47.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.650+0000 7f78e18b9640 1 -- 172.21.15.155:0/1639355042 learned_addr learned my addr 172.21.15.155:0/1639355042 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:35:47.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.651+0000 7f78d1ffb640 10 client.?.objecter ms_handle_connect 0x7f78dc1430c0 2024-03-25T18:35:47.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.651+0000 7f78d1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:47.653 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.652+0000 7f78d1ffb640 10 client.?.objecter ms_handle_connect 0x7f78dc0a98f0 2024-03-25T18:35:47.653 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.652+0000 7f78d1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:47.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.658+0000 7f78d1ffb640 1 -- 172.21.15.155:0/1639355042 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3316928127 0 0) 0x7f78dc1db930 con 0x7f78dc1430c0 2024-03-25T18:35:47.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.658+0000 7f78d1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:35:47.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.658+0000 7f78d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:47.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.658+0000 7f78d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:47.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.658+0000 7f78d1ffb640 10 monclient(hunting): my global_id is 4725 2024-03-25T18:35:47.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.658+0000 7f78d1ffb640 1 -- 172.21.15.155:0/1639355042 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78c0004eb0 con 0x7f78dc1430c0 2024-03-25T18:35:47.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.658+0000 7f78d1ffb640 10 client.?.objecter ms_handle_connect 0x7f78dc13f710 2024-03-25T18:35:47.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.659+0000 7f78d1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:47.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.667+0000 7f78d1ffb640 1 -- 172.21.15.155:0/1639355042 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 136893448 0 0) 0x7f78dc070480 con 0x7f78dc0a98f0 2024-03-25T18:35:47.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.667+0000 7f78d1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:35:47.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.667+0000 7f78d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:47.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.667+0000 7f78d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:47.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.667+0000 7f78d1ffb640 10 monclient(hunting): my global_id is 4736 2024-03-25T18:35:47.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.667+0000 7f78d1ffb640 1 -- 172.21.15.155:0/1639355042 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78dc1db930 con 0x7f78dc0a98f0 2024-03-25T18:35:47.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.670+0000 7f78d1ffb640 1 -- 172.21.15.155:0/1639355042 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1073498096 0 0) 0x7f78c0004eb0 con 0x7f78dc1430c0 2024-03-25T18:35:47.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.670+0000 7f78d1ffb640 1 -- 172.21.15.155:0/1639355042 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f78dc070480 con 0x7f78dc1430c0 2024-03-25T18:35:47.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.671+0000 7f78d1ffb640 1 -- 172.21.15.155:0/1639355042 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f78d4001c90 con 0x7f78dc1430c0 2024-03-25T18:35:47.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.671+0000 7f78d1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:47.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.671+0000 7f78d1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:35:47.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.671+0000 7f78d1ffb640 10 monclient(hunting): dump: 2024-03-25T18:35:47.672 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:35:47.672 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:35:47.672 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:47.672 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:47.672 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:47.672 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:35:47.673 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:35:47.673 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:35:47.673 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:35:47.673 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:35:47.673 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.671+0000 7f78d1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:35:47.678 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.677+0000 7f78d1ffb640 1 -- 172.21.15.155:0/1639355042 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 151046766 0 0) 0x7f78dc070210 con 0x7f78dc13f710 2024-03-25T18:35:47.678 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.677+0000 7f78d1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:35:47.678 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.677+0000 7f78d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:47.678 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.677+0000 7f78d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:47.678 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.677+0000 7f78d1ffb640 10 monclient(hunting): my global_id is 4723 2024-03-25T18:35:47.678 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.677+0000 7f78d1ffb640 1 -- 172.21.15.155:0/1639355042 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78c0004eb0 con 0x7f78dc13f710 2024-03-25T18:35:47.684 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.683+0000 7f78d1ffb640 1 -- 172.21.15.155:0/1639355042 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1681893479 0 0) 0x7f78c0004eb0 con 0x7f78dc13f710 2024-03-25T18:35:47.684 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.683+0000 7f78d1ffb640 1 -- 172.21.15.155:0/1639355042 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f78dc070210 con 0x7f78dc13f710 2024-03-25T18:35:47.684 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.683+0000 7f78d1ffb640 1 -- 172.21.15.155:0/1639355042 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f78cc001b00 con 0x7f78dc13f710 2024-03-25T18:35:47.684 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.683+0000 7f78d1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:47.684 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.683+0000 7f78d1ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:35:47.684 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.683+0000 7f78d1ffb640 10 monclient(hunting): dump: 2024-03-25T18:35:47.684 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:35:47.684 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:35:47.684 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:47.684 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:47.685 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:47.685 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:35:47.685 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:35:47.685 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:35:47.685 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:35:47.685 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:35:47.689 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.687+0000 7f78d1ffb640 1 -- 172.21.15.155:0/1639355042 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2541739430 0 0) 0x7f78dc070480 con 0x7f78dc1430c0 2024-03-25T18:35:47.689 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.687+0000 7f78d1ffb640 1 -- 172.21.15.155:0/1639355042 >> v1:172.21.15.155:6790/0 conn(0x7f78dc0a98f0 legacy=0x7f78dc06ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:47.689 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.687+0000 7f78d1ffb640 1 -- 172.21.15.155:0/1639355042 >> v1:172.21.15.167:6789/0 conn(0x7f78dc13f710 legacy=0x7f78dc1dbdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:47.689 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.687+0000 7f78d1ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:35:47.689 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.687+0000 7f78d1ffb640 1 monclient: found mon.a 2024-03-25T18:35:47.689 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.687+0000 7f78d1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:47.689 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.687+0000 7f78d1ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:35:47.689 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.687+0000 7f78d1ffb640 1 -- 172.21.15.155:0/1639355042 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78dc06f2c0 con 0x7f78dc1430c0 2024-03-25T18:35:47.689 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.687+0000 7f78d1ffb640 10 monclient: _finish_auth 0 2024-03-25T18:35:47.689 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.687+0000 7f78d1ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:35:47.689 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.687+0000 7f78d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:47.690 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.687+0000 7f78e331c640 5 monclient: authenticate success, global_id 4725 2024-03-25T18:35:47.690 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.687+0000 7f78e331c640 10 monclient: _renew_subs 2024-03-25T18:35:47.690 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.687+0000 7f78e331c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:35:47.690 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.687+0000 7f78e331c640 1 -- 172.21.15.155:0/1639355042 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78dc06e5d0 con 0x7f78dc1430c0 2024-03-25T18:35:47.690 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.687+0000 7f78e331c640 10 client.4725.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:35:47.690 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.688+0000 7f78e331c640 10 monclient: _renew_subs 2024-03-25T18:35:47.690 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.688+0000 7f78e331c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:35:47.690 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.688+0000 7f78e331c640 1 -- 172.21.15.155:0/1639355042 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f78dc139f00 con 0x7f78dc1430c0 2024-03-25T18:35:47.690 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.688+0000 7f78e331c640 1 librados: init done 2024-03-25T18:35:47.691 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.688+0000 7f78e331c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:47.691 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.688+0000 7f78e331c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:47.691 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.688+0000 7f78e331c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:35:47.691 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.688+0000 7f78e331c640 1 -- 172.21.15.155:0/1639355042 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78dc0627b0 con 0x7f78dc1430c0 2024-03-25T18:35:47.721 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.719+0000 7f78d1ffb640 1 -- 172.21.15.155:0/1639355042 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f78d4003970 con 0x7f78dc1430c0 2024-03-25T18:35:47.721 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.719+0000 7f78d1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:47.721 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.719+0000 7f78d1ffb640 1 -- 172.21.15.155:0/1639355042 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f78d4003be0 con 0x7f78dc1430c0 2024-03-25T18:35:47.721 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.719+0000 7f78d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:47.721 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.719+0000 7f78d1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:35:47.721 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.719+0000 7f78d1ffb640 10 monclient: dump: 2024-03-25T18:35:47.721 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:35:47.721 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:35:47.721 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:47.721 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:47.721 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:47.721 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:35:47.721 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:35:47.722 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:35:47.722 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:35:47.722 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:35:47.722 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.719+0000 7f78d37fe640 4 set_mon_vals no callback set 2024-03-25T18:35:47.722 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.720+0000 7f78d1ffb640 1 -- 172.21.15.155:0/1639355042 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7f78d4003e00 con 0x7f78dc1430c0 2024-03-25T18:35:47.722 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.721+0000 7f78e10b8640 10 monclient: get_auth_request con 0x7f78c006dfb0 auth_method 0 2024-03-25T18:35:47.722 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.721+0000 7f78d1ffb640 10 client.4725.objecter ms_handle_connect 0x7f78c006dfb0 2024-03-25T18:35:47.726 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.725+0000 7f78d1ffb640 1 -- 172.21.15.155:0/1639355042 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1555029372 0 0) 0x7f78d4054920 con 0x7f78dc1430c0 2024-03-25T18:35:47.726 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.725+0000 7f78d1ffb640 10 client.4725.objecter ms_dispatch 0x7f78dc000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:35:47.726 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.725+0000 7f78d1ffb640 3 client.4725.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:35:47.726 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.725+0000 7f78d1ffb640 3 client.4725.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:35:47.727 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:47.725+0000 7f78d1ffb640 20 client.4725.objecter dump_active .. 0 homeless 2024-03-25T18:35:48.006 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:48.005+0000 7f78d1ffb640 1 -- 172.21.15.155:0/1639355042 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f78d4050430 con 0x7f78dc1430c0 2024-03-25T18:35:48.006 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:48.005+0000 7f78d1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:48.007 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:48.005+0000 7f78d1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:35:48.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:48.258+0000 7f78e331c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:48.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:48.258+0000 7f78e331c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:48.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:48.258+0000 7f78e331c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-25T18:35:48.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:48.258+0000 7f78e331c640 1 -- 172.21.15.155:0/1639355042 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f78dc074cc0 con 0x7f78dc1430c0 2024-03-25T18:35:48.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:48.278+0000 7f78d1ffb640 1 -- 172.21.15.155:0/1639355042 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3170230695) 0x7f78d4052360 con 0x7f78dc1430c0 2024-03-25T18:35:48.280 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:48.278+0000 7f78d1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:48.280 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:48.278+0000 7f78d1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:35:48.280 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:35:48.280 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":11,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:34:47.433406+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:34:47.433406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:35:48.282 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:48.281+0000 7f78e331c640 10 librados: watch_flush enter 2024-03-25T18:35:48.282 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:48.281+0000 7f78e331c640 10 librados: watch_flush exit 2024-03-25T18:35:48.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:48.281+0000 7f78e331c640 20 client.4725.objecter shutdown clearing up homeless session... 2024-03-25T18:35:48.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:48.281+0000 7f78e331c640 10 client.4725.objecter successfully canceled tick 2024-03-25T18:35:48.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:48.281+0000 7f78e331c640 1 -- 172.21.15.155:0/1639355042 >> v1:172.21.15.167:6800/2919070503 conn(0x7f78c006dfb0 legacy=0x7f78c0070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:48.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:48.281+0000 7f78e331c640 10 monclient: shutdown 2024-03-25T18:35:48.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:48.281+0000 7f78e331c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:48.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:48.281+0000 7f78e331c640 1 -- 172.21.15.155:0/1639355042 >> v1:172.21.15.155:6789/0 conn(0x7f78dc1430c0 legacy=0x7f78dc1e6280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:48.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:48.281+0000 7f78e331c640 1 -- 172.21.15.155:0/1639355042 shutdown_connections 2024-03-25T18:35:48.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:48.281+0000 7f78e331c640 1 -- 172.21.15.155:0/1639355042 >> 172.21.15.155:0/1639355042 conn(0x7f78dc0b7ea0 msgr2=0x7f78dc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:48.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:48.281+0000 7f78e331c640 1 -- 172.21.15.155:0/1639355042 shutdown_connections 2024-03-25T18:35:48.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:48.281+0000 7f78e331c640 1 -- 172.21.15.155:0/1639355042 wait complete. 2024-03-25T18:35:48.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:48.281+0000 7f78e331c640 1 librados: shutdown 2024-03-25T18:35:48.292 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:35:49.545 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:11:23.149 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:49.557 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:11:23.161 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:49.566 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:11:23.171 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:49.568 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:11:23.146 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:49.571 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:11:23.176 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:49.571 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:11:23.150 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:49.583 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:11:23.187 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:49.592 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:11:23.171 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:49.592 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:11:23.197 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:52.775 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:11:26.379 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:52.819 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:11:26.398 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:52.831 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:11:26.436 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:52.831 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:11:26.410 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:52.859 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:11:26.438 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:52.860 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:11:26.465 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:52.886 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:11:26.491 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:52.903 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:11:26.508 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:52.924 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:11:26.529 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:55.099 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:35:55.211 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.210+0000 7f048b225640 10 monclient: get_monmap_and_config 2024-03-25T18:35:55.211 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.210+0000 7f048b225640 10 monclient: build_initial_monmap 2024-03-25T18:35:55.211 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.210+0000 7f048b225640 10 monclient: monmap: 2024-03-25T18:35:55.211 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:35:55.211 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:35:55.211 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:35:55.211260+0000 2024-03-25T18:35:55.211 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:35:55.211260+0000 2024-03-25T18:35:55.211 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:55.211 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:35:55.212 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:35:55.212 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:35:55.212 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:35:55.212 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:35:55.213 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.211+0000 7f048b225640 1 Processor -- start 2024-03-25T18:35:55.213 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.211+0000 7f048b225640 1 -- start start 2024-03-25T18:35:55.213 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.211+0000 7f048b225640 10 monclient: init 2024-03-25T18:35:55.213 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.211+0000 7f048b225640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:55.213 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.211+0000 7f048b225640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:35:55.213 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.211+0000 7f048b225640 10 monclient(hunting): picked mon.noname-b con 0x7f04840b4d60 addr v1:172.21.15.155:6790/0 2024-03-25T18:35:55.213 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.211+0000 7f048b225640 10 monclient(hunting): picked mon.noname-c con 0x7f04840b1730 addr v1:172.21.15.167:6789/0 2024-03-25T18:35:55.213 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.211+0000 7f048b225640 10 monclient(hunting): picked mon.noname-a con 0x7f048414be90 addr v1:172.21.15.155:6789/0 2024-03-25T18:35:55.213 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.211+0000 7f048b225640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04840af940 con 0x7f048414be90 2024-03-25T18:35:55.214 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.211+0000 7f048b225640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04840b0a70 con 0x7f04840b1730 2024-03-25T18:35:55.214 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.211+0000 7f048b225640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04840b0d50 con 0x7f04840b4d60 2024-03-25T18:35:55.214 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.211+0000 7f048b225640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:55.214 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.211+0000 7f048b225640 10 monclient(hunting): authenticate will time out at 8201.521400s 2024-03-25T18:35:55.214 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.212+0000 7f047bfff640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f04840b1730 0x7f04840b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:45266/0 (socket says 172.21.15.155:45266) 2024-03-25T18:35:55.214 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.212+0000 7f047bfff640 1 -- 172.21.15.155:0/4147361928 learned_addr learned my addr 172.21.15.155:0/4147361928 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:35:55.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.274+0000 7f047b7fe640 1 -- 172.21.15.155:0/4147361928 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1434764956 0 0) 0x7f04840af940 con 0x7f048414be90 2024-03-25T18:35:55.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.274+0000 7f047b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:35:55.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.274+0000 7f047b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:55.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.274+0000 7f047b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:55.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.274+0000 7f047b7fe640 10 monclient(hunting): my global_id is 4728 2024-03-25T18:35:55.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.274+0000 7f047b7fe640 1 -- 172.21.15.155:0/4147361928 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f046c003520 con 0x7f048414be90 2024-03-25T18:35:55.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.281+0000 7f047b7fe640 1 -- 172.21.15.155:0/4147361928 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4102611881 0 0) 0x7f046c003520 con 0x7f048414be90 2024-03-25T18:35:55.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.281+0000 7f047b7fe640 1 -- 172.21.15.155:0/4147361928 >> v1:172.21.15.155:6790/0 conn(0x7f04840b4d60 legacy=0x7f048413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:55.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.281+0000 7f047b7fe640 1 -- 172.21.15.155:0/4147361928 >> v1:172.21.15.167:6789/0 conn(0x7f04840b1730 legacy=0x7f04840b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:55.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.281+0000 7f047b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:35:55.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.281+0000 7f047b7fe640 1 monclient: found mon.noname-a 2024-03-25T18:35:55.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.281+0000 7f047b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:55.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.281+0000 7f047b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-25T18:35:55.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.281+0000 7f047b7fe640 1 -- 172.21.15.155:0/4147361928 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04840b1030 con 0x7f048414be90 2024-03-25T18:35:55.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.281+0000 7f047b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:35:55.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.281+0000 7f047b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:35:55.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.281+0000 7f047b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:55.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.281+0000 7f047b7fe640 1 -- 172.21.15.155:0/4147361928 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f047c001a10 con 0x7f048414be90 2024-03-25T18:35:55.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.281+0000 7f047b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:55.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.281+0000 7f047b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:35:55.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.281+0000 7f047b7fe640 10 monclient: dump: 2024-03-25T18:35:55.284 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:35:55.284 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:35:55.284 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:55.284 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:55.284 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:55.284 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:35:55.284 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:35:55.284 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:35:55.284 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:35:55.284 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:35:55.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.281+0000 7f048b225640 5 monclient: authenticate success, global_id 4728 2024-03-25T18:35:55.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.281+0000 7f048b225640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:55.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.286+0000 7f047b7fe640 1 -- 172.21.15.155:0/4147361928 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f047c001b70 con 0x7f048414be90 2024-03-25T18:35:55.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.286+0000 7f047b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:55.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.286+0000 7f048b225640 10 monclient: get_monmap_and_config success 2024-03-25T18:35:55.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.286+0000 7f048b225640 4 set_mon_vals no callback set 2024-03-25T18:35:55.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.287+0000 7f048b225640 10 monclient: shutdown 2024-03-25T18:35:55.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.287+0000 7f048b225640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:55.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.287+0000 7f048b225640 1 -- 172.21.15.155:0/4147361928 >> v1:172.21.15.155:6789/0 conn(0x7f048414be90 legacy=0x7f048414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:55.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.287+0000 7f048b225640 1 -- 172.21.15.155:0/4147361928 shutdown_connections 2024-03-25T18:35:55.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.287+0000 7f048b225640 1 -- 172.21.15.155:0/4147361928 >> 172.21.15.155:0/4147361928 conn(0x7f04840617a0 msgr2=0x7f0484061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:55.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.287+0000 7f048b225640 1 -- 172.21.15.155:0/4147361928 shutdown_connections 2024-03-25T18:35:55.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.287+0000 7f048b225640 1 -- 172.21.15.155:0/4147361928 wait complete. 2024-03-25T18:35:55.289 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.288+0000 7f048b225640 10 monclient: build_initial_monmap 2024-03-25T18:35:55.289 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.288+0000 7f048b225640 10 monclient: monmap: 2024-03-25T18:35:55.289 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:35:55.289 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:35:55.289 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:35:55.289 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:35:55.290 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:55.290 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:35:55.290 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:35:55.290 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:35:55.290 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:35:55.290 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:35:55.290 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.288+0000 7f048b225640 1 librados: starting msgr at 2024-03-25T18:35:55.290 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.288+0000 7f048b225640 1 librados: starting objecter 2024-03-25T18:35:55.290 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.288+0000 7f048b225640 1 Processor -- start 2024-03-25T18:35:55.290 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.288+0000 7f048b225640 1 -- start start 2024-03-25T18:35:55.290 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.288+0000 7f048b225640 1 librados: setting wanted keys 2024-03-25T18:35:55.290 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.288+0000 7f048b225640 1 librados: calling monclient init 2024-03-25T18:35:55.290 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.288+0000 7f048b225640 10 monclient: init 2024-03-25T18:35:55.291 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.289+0000 7f048b225640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:55.291 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.289+0000 7f048b225640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:35:55.291 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.289+0000 7f048b225640 10 monclient(hunting): picked mon.noname-b con 0x7f04840b1730 addr v1:172.21.15.167:6789/0 2024-03-25T18:35:55.292 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.289+0000 7f048b225640 10 monclient(hunting): picked mon.noname-a con 0x7f04840b4d60 addr v1:172.21.15.155:6789/0 2024-03-25T18:35:55.292 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.289+0000 7f048b225640 10 monclient(hunting): picked mon.noname-c con 0x7f048414be90 addr v1:172.21.15.155:6790/0 2024-03-25T18:35:55.292 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.289+0000 7f048b225640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04841d7c80 con 0x7f04840b4d60 2024-03-25T18:35:55.292 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.289+0000 7f048b225640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04840aad20 con 0x7f04840b1730 2024-03-25T18:35:55.292 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.289+0000 7f048b225640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04841d78a0 con 0x7f048414be90 2024-03-25T18:35:55.292 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.289+0000 7f048b225640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:55.292 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.289+0000 7f048b225640 10 monclient(hunting): authenticate will time out at 8201.599391s 2024-03-25T18:35:55.299 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.298+0000 7f047bfff640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f04840b4d60 0x7f04841d8110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:37264/0 (socket says 172.21.15.155:37264) 2024-03-25T18:35:55.299 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.298+0000 7f047bfff640 1 -- 172.21.15.155:0/3814293979 learned_addr learned my addr 172.21.15.155:0/3814293979 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:35:55.304 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.303+0000 7f04797fa640 10 client.?.objecter ms_handle_connect 0x7f04840b4d60 2024-03-25T18:35:55.304 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.303+0000 7f04797fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:55.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.307+0000 7f04797fa640 10 client.?.objecter ms_handle_connect 0x7f04840b1730 2024-03-25T18:35:55.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.307+0000 7f04797fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:55.313 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.312+0000 7f04797fa640 10 client.?.objecter ms_handle_connect 0x7f048414be90 2024-03-25T18:35:55.313 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.312+0000 7f04797fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:55.314 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.314+0000 7f04797fa640 1 -- 172.21.15.155:0/3814293979 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 425123141 0 0) 0x7f04840aad20 con 0x7f04840b1730 2024-03-25T18:35:55.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.314+0000 7f04797fa640 10 monclient(hunting): _negotiate 2024-03-25T18:35:55.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.314+0000 7f04797fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:55.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.314+0000 7f04797fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:55.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.314+0000 7f04797fa640 10 monclient(hunting): my global_id is 4729 2024-03-25T18:35:55.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.314+0000 7f04797fa640 1 -- 172.21.15.155:0/3814293979 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0470004eb0 con 0x7f04840b1730 2024-03-25T18:35:55.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.317+0000 7f04797fa640 1 -- 172.21.15.155:0/3814293979 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 962438325 0 0) 0x7f04841d7c80 con 0x7f04840b4d60 2024-03-25T18:35:55.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.317+0000 7f04797fa640 10 monclient(hunting): _negotiate 2024-03-25T18:35:55.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.317+0000 7f04797fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:55.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.317+0000 7f04797fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:55.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.317+0000 7f04797fa640 10 monclient(hunting): my global_id is 4731 2024-03-25T18:35:55.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.317+0000 7f04797fa640 1 -- 172.21.15.155:0/3814293979 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04840aad20 con 0x7f04840b4d60 2024-03-25T18:35:55.326 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.325+0000 7f04797fa640 1 -- 172.21.15.155:0/3814293979 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1299662521 0 0) 0x7f04840aad20 con 0x7f04840b4d60 2024-03-25T18:35:55.326 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.325+0000 7f04797fa640 1 -- 172.21.15.155:0/3814293979 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04841d7c80 con 0x7f04840b4d60 2024-03-25T18:35:55.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.329+0000 7f04797fa640 1 -- 172.21.15.155:0/3814293979 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2036425401 0 0) 0x7f0470004eb0 con 0x7f04840b1730 2024-03-25T18:35:55.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.329+0000 7f04797fa640 1 -- 172.21.15.155:0/3814293979 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04840aad20 con 0x7f04840b1730 2024-03-25T18:35:55.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.329+0000 7f04797fa640 1 -- 172.21.15.155:0/3814293979 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f04800018c0 con 0x7f04840b1730 2024-03-25T18:35:55.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.329+0000 7f04797fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:55.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.329+0000 7f04797fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:35:55.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.329+0000 7f04797fa640 10 monclient(hunting): dump: 2024-03-25T18:35:55.330 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:35:55.330 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:35:55.330 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:55.330 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:55.330 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:55.330 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:35:55.331 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:35:55.331 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:35:55.331 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:35:55.331 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:35:55.331 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.329+0000 7f04797fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:35:55.333 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.332+0000 7f04797fa640 1 -- 172.21.15.155:0/3814293979 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1849619658 0 0) 0x7f04841d78a0 con 0x7f048414be90 2024-03-25T18:35:55.333 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.332+0000 7f04797fa640 10 monclient(hunting): _negotiate 2024-03-25T18:35:55.333 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.332+0000 7f04797fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:55.333 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.332+0000 7f04797fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:55.334 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.332+0000 7f04797fa640 10 monclient(hunting): my global_id is 4742 2024-03-25T18:35:55.334 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.332+0000 7f04797fa640 1 -- 172.21.15.155:0/3814293979 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0470004eb0 con 0x7f048414be90 2024-03-25T18:35:55.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.351+0000 7f04797fa640 1 -- 172.21.15.155:0/3814293979 <== mon.1 v1:172.21.15.167:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1130523715 0 0) 0x7f04840aad20 con 0x7f04840b1730 2024-03-25T18:35:55.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.351+0000 7f04797fa640 1 -- 172.21.15.155:0/3814293979 >> v1:172.21.15.155:6790/0 conn(0x7f048414be90 legacy=0x7f0484137e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:55.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.351+0000 7f04797fa640 1 -- 172.21.15.155:0/3814293979 >> v1:172.21.15.155:6789/0 conn(0x7f04840b4d60 legacy=0x7f04841d8110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:55.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.351+0000 7f04797fa640 10 monclient: _finish_hunting 0 2024-03-25T18:35:55.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.351+0000 7f04797fa640 1 monclient: found mon.b 2024-03-25T18:35:55.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.351+0000 7f04797fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:55.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.351+0000 7f04797fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:35:55.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.351+0000 7f04797fa640 1 -- 172.21.15.155:0/3814293979 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0484138580 con 0x7f04840b1730 2024-03-25T18:35:55.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.351+0000 7f04797fa640 10 monclient: _finish_auth 0 2024-03-25T18:35:55.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.351+0000 7f04797fa640 10 monclient: _check_auth_tickets 2024-03-25T18:35:55.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.351+0000 7f04797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:55.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.351+0000 7f048b225640 5 monclient: authenticate success, global_id 4729 2024-03-25T18:35:55.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.351+0000 7f048b225640 10 monclient: _renew_subs 2024-03-25T18:35:55.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.351+0000 7f048b225640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:35:55.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.351+0000 7f048b225640 1 -- 172.21.15.155:0/3814293979 --> v1:172.21.15.167:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0484138830 con 0x7f04840b1730 2024-03-25T18:35:55.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.351+0000 7f048b225640 10 client.4729.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:35:55.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.351+0000 7f048b225640 10 monclient: _renew_subs 2024-03-25T18:35:55.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.351+0000 7f048b225640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:35:55.354 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.351+0000 7f048b225640 1 -- 172.21.15.155:0/3814293979 --> v1:172.21.15.167:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f04840a98c0 con 0x7f04840b1730 2024-03-25T18:35:55.354 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.351+0000 7f048b225640 1 librados: init done 2024-03-25T18:35:55.354 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.352+0000 7f048b225640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:55.354 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.352+0000 7f048b225640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:55.354 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.352+0000 7f048b225640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:35:55.354 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.352+0000 7f048b225640 1 -- 172.21.15.155:0/3814293979 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04841d7a30 con 0x7f04840b1730 2024-03-25T18:35:55.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.634+0000 7f04797fa640 1 -- 172.21.15.155:0/3814293979 <== mon.1 v1:172.21.15.167:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04800035f0 con 0x7f04840b1730 2024-03-25T18:35:55.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.634+0000 7f04797fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:55.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.634+0000 7f04797fa640 1 -- 172.21.15.155:0/3814293979 <== mon.1 v1:172.21.15.167:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f0480003900 con 0x7f04840b1730 2024-03-25T18:35:55.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.634+0000 7f04797fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:55.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.634+0000 7f04797fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:35:55.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.634+0000 7f04797fa640 10 monclient: dump: 2024-03-25T18:35:55.636 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:35:55.636 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:35:55.636 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:55.637 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:35:55.637 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:55.637 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:35:55.637 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:35:55.637 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:35:55.637 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:35:55.637 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:35:55.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.634+0000 7f047affd640 4 set_mon_vals no callback set 2024-03-25T18:35:55.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.634+0000 7f04797fa640 1 -- 172.21.15.155:0/3814293979 <== mon.1 v1:172.21.15.167:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7f04800198d0 con 0x7f04840b1730 2024-03-25T18:35:55.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.635+0000 7f047bfff640 10 monclient: get_auth_request con 0x7f047006e050 auth_method 0 2024-03-25T18:35:55.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.635+0000 7f04797fa640 10 client.4729.objecter ms_handle_connect 0x7f047006e050 2024-03-25T18:35:55.642 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.641+0000 7f04797fa640 1 -- 172.21.15.155:0/3814293979 <== mon.1 v1:172.21.15.167:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1555029372 0 0) 0x7f04800535c0 con 0x7f04840b1730 2024-03-25T18:35:55.642 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.641+0000 7f04797fa640 10 client.4729.objecter ms_dispatch 0x7f0484000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:35:55.642 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.641+0000 7f04797fa640 3 client.4729.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:35:55.642 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.641+0000 7f04797fa640 3 client.4729.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:35:55.642 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.641+0000 7f04797fa640 20 client.4729.objecter dump_active .. 0 homeless 2024-03-25T18:35:55.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.644+0000 7f04797fa640 1 -- 172.21.15.155:0/3814293979 <== mon.1 v1:172.21.15.167:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f048004ef00 con 0x7f04840b1730 2024-03-25T18:35:55.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.644+0000 7f04797fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:55.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.644+0000 7f04797fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:35:55.892 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.891+0000 7f048b225640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:55.892 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.891+0000 7f048b225640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:55.892 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.891+0000 7f048b225640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:35:55.892 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.891+0000 7f048b225640 1 -- 172.21.15.155:0/3814293979 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f04840627b0 con 0x7f04840b1730 2024-03-25T18:35:55.923 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.922+0000 7f04797fa640 1 -- 172.21.15.155:0/3814293979 <== mon.1 v1:172.21.15.167:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3170230695) 0x7f0480051000 con 0x7f04840b1730 2024-03-25T18:35:55.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.922+0000 7f04797fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:55.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.922+0000 7f04797fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:35:55.924 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:35:55.924 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":11,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:34:47.433406+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:34:47.433406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:35:55.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.925+0000 7f048b225640 10 librados: watch_flush enter 2024-03-25T18:35:55.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.925+0000 7f048b225640 10 librados: watch_flush exit 2024-03-25T18:35:55.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.925+0000 7f048b225640 20 client.4729.objecter shutdown clearing up homeless session... 2024-03-25T18:35:55.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.925+0000 7f048b225640 10 client.4729.objecter successfully canceled tick 2024-03-25T18:35:55.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.925+0000 7f048b225640 1 -- 172.21.15.155:0/3814293979 >> v1:172.21.15.167:6800/2919070503 conn(0x7f047006e050 legacy=0x7f0470070510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:55.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.925+0000 7f048b225640 10 monclient: shutdown 2024-03-25T18:35:55.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.925+0000 7f048b225640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:55.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.925+0000 7f048b225640 1 -- 172.21.15.155:0/3814293979 >> v1:172.21.15.167:6789/0 conn(0x7f04840b1730 legacy=0x7f0484143190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:55.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.925+0000 7f048b225640 1 -- 172.21.15.155:0/3814293979 shutdown_connections 2024-03-25T18:35:55.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.925+0000 7f048b225640 1 -- 172.21.15.155:0/3814293979 >> 172.21.15.155:0/3814293979 conn(0x7f04840617a0 msgr2=0x7f04840b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:55.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.925+0000 7f048b225640 1 -- 172.21.15.155:0/3814293979 shutdown_connections 2024-03-25T18:35:55.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.925+0000 7f048b225640 1 -- 172.21.15.155:0/3814293979 wait complete. 2024-03-25T18:35:55.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:35:55.926+0000 7f048b225640 1 librados: shutdown 2024-03-25T18:35:55.938 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:36:02.744 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:36:02.856 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.856+0000 7f081a3ed640 10 monclient: get_monmap_and_config 2024-03-25T18:36:02.857 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.856+0000 7f081a3ed640 10 monclient: build_initial_monmap 2024-03-25T18:36:02.857 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.856+0000 7f081a3ed640 10 monclient: monmap: 2024-03-25T18:36:02.857 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:36:02.857 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:36:02.857 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:36:02.856998+0000 2024-03-25T18:36:02.857 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:36:02.856998+0000 2024-03-25T18:36:02.857 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:02.857 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:36:02.857 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:36:02.857 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:36:02.857 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:36:02.857 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:36:02.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.856+0000 7f081a3ed640 1 Processor -- start 2024-03-25T18:36:02.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.857+0000 7f081a3ed640 1 -- start start 2024-03-25T18:36:02.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.857+0000 7f081a3ed640 10 monclient: init 2024-03-25T18:36:02.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.857+0000 7f081a3ed640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:02.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.857+0000 7f081a3ed640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:36:02.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.857+0000 7f081a3ed640 10 monclient(hunting): picked mon.noname-a con 0x7f0814061830 addr v1:172.21.15.155:6789/0 2024-03-25T18:36:02.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.857+0000 7f081a3ed640 10 monclient(hunting): picked mon.noname-c con 0x7f0814146060 addr v1:172.21.15.167:6789/0 2024-03-25T18:36:02.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.857+0000 7f081a3ed640 10 monclient(hunting): picked mon.noname-b con 0x7f081413f7e0 addr v1:172.21.15.155:6790/0 2024-03-25T18:36:02.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.857+0000 7f081a3ed640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f081406f380 con 0x7f0814061830 2024-03-25T18:36:02.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.857+0000 7f081a3ed640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f081406f5f0 con 0x7f0814146060 2024-03-25T18:36:02.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.857+0000 7f081a3ed640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f081406f8d0 con 0x7f081413f7e0 2024-03-25T18:36:02.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.857+0000 7f081a3ed640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:02.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.857+0000 7f081a3ed640 10 monclient(hunting): authenticate will time out at 8209.167490s 2024-03-25T18:36:02.866 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.865+0000 7f08137fe640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f0814146060 0x7f081413f0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:44554/0 (socket says 172.21.15.155:44554) 2024-03-25T18:36:02.866 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.865+0000 7f08137fe640 1 -- 172.21.15.155:0/1296333158 learned_addr learned my addr 172.21.15.155:0/1296333158 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:36:02.877 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.876+0000 7f0812ffd640 1 -- 172.21.15.155:0/1296333158 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2787541923 0 0) 0x7f081406f5f0 con 0x7f0814146060 2024-03-25T18:36:02.877 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.876+0000 7f0812ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:36:02.877 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.876+0000 7f0812ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:02.877 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.876+0000 7f0812ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:02.877 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.876+0000 7f0812ffd640 10 monclient(hunting): my global_id is 4732 2024-03-25T18:36:02.878 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.876+0000 7f0812ffd640 1 -- 172.21.15.155:0/1296333158 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07fc003520 con 0x7f0814146060 2024-03-25T18:36:02.879 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.878+0000 7f0812ffd640 1 -- 172.21.15.155:0/1296333158 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1666809248 0 0) 0x7f081406f380 con 0x7f0814061830 2024-03-25T18:36:02.879 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.878+0000 7f0812ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:36:02.880 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.878+0000 7f0812ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:02.880 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.878+0000 7f0812ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:02.880 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.878+0000 7f0812ffd640 10 monclient(hunting): my global_id is 4734 2024-03-25T18:36:02.880 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.878+0000 7f0812ffd640 1 -- 172.21.15.155:0/1296333158 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f081406f5f0 con 0x7f0814061830 2024-03-25T18:36:02.885 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.884+0000 7f0812ffd640 1 -- 172.21.15.155:0/1296333158 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1992599542 0 0) 0x7f07fc003520 con 0x7f0814146060 2024-03-25T18:36:02.885 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.884+0000 7f0812ffd640 1 -- 172.21.15.155:0/1296333158 >> v1:172.21.15.155:6790/0 conn(0x7f081413f7e0 legacy=0x7f081414dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:02.885 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.884+0000 7f0812ffd640 1 -- 172.21.15.155:0/1296333158 >> v1:172.21.15.155:6789/0 conn(0x7f0814061830 legacy=0x7f0814145950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:02.885 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.884+0000 7f0812ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:36:02.885 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.884+0000 7f0812ffd640 1 monclient: found mon.noname-c 2024-03-25T18:36:02.885 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.884+0000 7f0812ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:02.885 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.884+0000 7f0812ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.167:6789/0 2024-03-25T18:36:02.885 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.884+0000 7f0812ffd640 1 -- 172.21.15.155:0/1296333158 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08140b0a30 con 0x7f0814146060 2024-03-25T18:36:02.885 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.884+0000 7f0812ffd640 10 monclient: _finish_auth 0 2024-03-25T18:36:02.885 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.884+0000 7f0812ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:36:02.885 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.884+0000 7f0812ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:02.886 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.884+0000 7f0812ffd640 1 -- 172.21.15.155:0/1296333158 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f0808001690 con 0x7f0814146060 2024-03-25T18:36:02.886 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.884+0000 7f0812ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:02.886 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.884+0000 7f0812ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:36:02.886 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.884+0000 7f0812ffd640 10 monclient: dump: 2024-03-25T18:36:02.886 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:36:02.886 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:36:02.886 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:36:02.886 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:36:02.886 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:02.886 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:36:02.886 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:36:02.886 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:36:02.886 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:36:02.886 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:36:02.886 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.884+0000 7f081a3ed640 5 monclient: authenticate success, global_id 4732 2024-03-25T18:36:02.887 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.884+0000 7f081a3ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:36:02.898 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.897+0000 7f0812ffd640 1 -- 172.21.15.155:0/1296333158 <== mon.1 v1:172.21.15.167:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f08080017f0 con 0x7f0814146060 2024-03-25T18:36:02.899 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.897+0000 7f0812ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:02.899 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.897+0000 7f0812ffd640 1 -- 172.21.15.155:0/1296333158 <== mon.1 v1:172.21.15.167:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f0808001b80 con 0x7f0814146060 2024-03-25T18:36:02.899 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.897+0000 7f0812ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:02.899 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.897+0000 7f0812ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:36:02.899 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.897+0000 7f0812ffd640 10 monclient: dump: 2024-03-25T18:36:02.899 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:36:02.899 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:36:02.899 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:36:02.899 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:36:02.899 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:02.899 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:36:02.899 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:36:02.899 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:36:02.899 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:36:02.900 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:36:02.900 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.897+0000 7f081a3ed640 10 monclient: get_monmap_and_config success 2024-03-25T18:36:02.900 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.897+0000 7f081a3ed640 4 set_mon_vals no callback set 2024-03-25T18:36:02.901 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.898+0000 7f081a3ed640 10 monclient: shutdown 2024-03-25T18:36:02.901 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.898+0000 7f081a3ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:02.901 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.898+0000 7f081a3ed640 1 -- 172.21.15.155:0/1296333158 >> v1:172.21.15.167:6789/0 conn(0x7f0814146060 legacy=0x7f081413f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:02.901 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.898+0000 7f081a3ed640 1 -- 172.21.15.155:0/1296333158 shutdown_connections 2024-03-25T18:36:02.901 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.898+0000 7f081a3ed640 1 -- 172.21.15.155:0/1296333158 >> 172.21.15.155:0/1296333158 conn(0x7f08140b11e0 msgr2=0x7f08140b15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:02.901 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.898+0000 7f081a3ed640 1 -- 172.21.15.155:0/1296333158 shutdown_connections 2024-03-25T18:36:02.901 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.898+0000 7f081a3ed640 1 -- 172.21.15.155:0/1296333158 wait complete. 2024-03-25T18:36:02.902 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.899+0000 7f081a3ed640 10 monclient: build_initial_monmap 2024-03-25T18:36:02.902 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.899+0000 7f081a3ed640 10 monclient: monmap: 2024-03-25T18:36:02.902 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:36:02.902 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:36:02.902 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:36:02.902 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:36:02.902 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:02.902 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:36:02.902 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:36:02.902 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:36:02.902 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:36:02.902 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:36:02.902 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.900+0000 7f081a3ed640 1 librados: starting msgr at 2024-03-25T18:36:02.902 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.900+0000 7f081a3ed640 1 librados: starting objecter 2024-03-25T18:36:02.903 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.900+0000 7f081a3ed640 1 Processor -- start 2024-03-25T18:36:02.903 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.900+0000 7f081a3ed640 1 -- start start 2024-03-25T18:36:02.903 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.900+0000 7f081a3ed640 1 librados: setting wanted keys 2024-03-25T18:36:02.903 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.900+0000 7f081a3ed640 1 librados: calling monclient init 2024-03-25T18:36:02.903 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.900+0000 7f081a3ed640 10 monclient: init 2024-03-25T18:36:02.904 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.901+0000 7f081a3ed640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:02.904 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.901+0000 7f081a3ed640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:36:02.904 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.901+0000 7f081a3ed640 10 monclient(hunting): picked mon.noname-b con 0x7f0814061830 addr v1:172.21.15.167:6789/0 2024-03-25T18:36:02.905 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.901+0000 7f081a3ed640 10 monclient(hunting): picked mon.noname-a con 0x7f081413f7e0 addr v1:172.21.15.155:6789/0 2024-03-25T18:36:02.905 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.901+0000 7f081a3ed640 10 monclient(hunting): picked mon.noname-c con 0x7f0814146060 addr v1:172.21.15.155:6790/0 2024-03-25T18:36:02.905 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.901+0000 7f081a3ed640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0814146670 con 0x7f081413f7e0 2024-03-25T18:36:02.905 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.901+0000 7f081a3ed640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08141cc230 con 0x7f0814061830 2024-03-25T18:36:02.905 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.901+0000 7f081a3ed640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08140aa950 con 0x7f0814146060 2024-03-25T18:36:02.905 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.901+0000 7f081a3ed640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:02.905 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.901+0000 7f081a3ed640 10 monclient(hunting): authenticate will time out at 8209.211482s 2024-03-25T18:36:02.905 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.902+0000 7f0813fff640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f0814061830 0x7f08140aa240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:44562/0 (socket says 172.21.15.155:44562) 2024-03-25T18:36:02.905 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.902+0000 7f0813fff640 1 -- 172.21.15.155:0/2075847243 learned_addr learned my addr 172.21.15.155:0/2075847243 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:36:02.905 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.902+0000 7f0810ff9640 10 client.?.objecter ms_handle_connect 0x7f0814061830 2024-03-25T18:36:02.905 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.902+0000 7f0810ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:02.908 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.908+0000 7f0810ff9640 10 client.?.objecter ms_handle_connect 0x7f0814146060 2024-03-25T18:36:02.908 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.908+0000 7f0810ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:02.914 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.913+0000 7f0810ff9640 1 -- 172.21.15.155:0/2075847243 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4157415416 0 0) 0x7f08141cc230 con 0x7f0814061830 2024-03-25T18:36:02.914 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.913+0000 7f0810ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:36:02.914 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.913+0000 7f0810ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:02.914 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.913+0000 7f0810ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:02.914 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.913+0000 7f0810ff9640 10 monclient(hunting): my global_id is 4735 2024-03-25T18:36:02.914 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.913+0000 7f0810ff9640 1 -- 172.21.15.155:0/2075847243 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0800004eb0 con 0x7f0814061830 2024-03-25T18:36:02.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.915+0000 7f0810ff9640 1 -- 172.21.15.155:0/2075847243 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 465759072 0 0) 0x7f08140aa950 con 0x7f0814146060 2024-03-25T18:36:02.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.915+0000 7f0810ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:36:02.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.915+0000 7f0810ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:02.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.915+0000 7f0810ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:02.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.915+0000 7f0810ff9640 10 monclient(hunting): my global_id is 4748 2024-03-25T18:36:02.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.915+0000 7f0810ff9640 1 -- 172.21.15.155:0/2075847243 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08141cc230 con 0x7f0814146060 2024-03-25T18:36:02.919 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.919+0000 7f0810ff9640 10 client.?.objecter ms_handle_connect 0x7f081413f7e0 2024-03-25T18:36:02.920 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.919+0000 7f0810ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:02.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.929+0000 7f0810ff9640 1 -- 172.21.15.155:0/2075847243 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3581219995 0 0) 0x7f0814146670 con 0x7f081413f7e0 2024-03-25T18:36:02.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.929+0000 7f0810ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:36:02.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.929+0000 7f0810ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:02.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.929+0000 7f0810ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:02.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.929+0000 7f0810ff9640 10 monclient(hunting): my global_id is 4737 2024-03-25T18:36:02.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.929+0000 7f0810ff9640 1 -- 172.21.15.155:0/2075847243 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08140aa950 con 0x7f081413f7e0 2024-03-25T18:36:02.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.929+0000 7f0810ff9640 1 -- 172.21.15.155:0/2075847243 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3384194287 0 0) 0x7f0800004eb0 con 0x7f0814061830 2024-03-25T18:36:02.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.929+0000 7f0810ff9640 1 -- 172.21.15.155:0/2075847243 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0814146670 con 0x7f0814061830 2024-03-25T18:36:02.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.929+0000 7f0810ff9640 1 -- 172.21.15.155:0/2075847243 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f0804001a10 con 0x7f0814061830 2024-03-25T18:36:02.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.929+0000 7f0810ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:02.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.929+0000 7f0810ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:36:02.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.929+0000 7f0810ff9640 10 monclient(hunting): dump: 2024-03-25T18:36:02.931 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:36:02.931 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:36:02.931 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:36:02.931 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:36:02.931 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:02.931 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:36:02.932 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:36:02.932 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:36:02.932 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:36:02.932 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:36:02.932 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.929+0000 7f0810ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:36:02.937 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.936+0000 7f0810ff9640 1 -- 172.21.15.155:0/2075847243 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2810257554 0 0) 0x7f08141cc230 con 0x7f0814146060 2024-03-25T18:36:02.938 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.936+0000 7f0810ff9640 1 -- 172.21.15.155:0/2075847243 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0800004eb0 con 0x7f0814146060 2024-03-25T18:36:02.938 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.936+0000 7f0810ff9640 1 -- 172.21.15.155:0/2075847243 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f080c001ba0 con 0x7f0814146060 2024-03-25T18:36:02.938 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.936+0000 7f0810ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:02.938 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.936+0000 7f0810ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:36:02.938 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.936+0000 7f0810ff9640 10 monclient(hunting): dump: 2024-03-25T18:36:02.938 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:36:02.938 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:36:02.938 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:36:02.938 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:36:02.938 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:02.938 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:36:02.938 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:36:02.939 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:36:02.939 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:36:02.939 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:36:02.941 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.940+0000 7f0810ff9640 1 -- 172.21.15.155:0/2075847243 <== mon.1 v1:172.21.15.167:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1750983197 0 0) 0x7f0814146670 con 0x7f0814061830 2024-03-25T18:36:02.941 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.940+0000 7f0810ff9640 1 -- 172.21.15.155:0/2075847243 >> v1:172.21.15.155:6790/0 conn(0x7f0814146060 legacy=0x7f08140ad3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:02.942 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.940+0000 7f0810ff9640 1 -- 172.21.15.155:0/2075847243 >> v1:172.21.15.155:6789/0 conn(0x7f081413f7e0 legacy=0x7f08140acca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:02.942 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.940+0000 7f0810ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:36:02.942 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.940+0000 7f0810ff9640 1 monclient: found mon.b 2024-03-25T18:36:02.942 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.940+0000 7f0810ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:02.942 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.940+0000 7f0810ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:36:02.942 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.940+0000 7f0810ff9640 1 -- 172.21.15.155:0/2075847243 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08141df9e0 con 0x7f0814061830 2024-03-25T18:36:02.942 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.940+0000 7f0810ff9640 10 monclient: _finish_auth 0 2024-03-25T18:36:02.942 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.940+0000 7f0810ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:36:02.942 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.940+0000 7f0810ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:02.942 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.940+0000 7f081a3ed640 5 monclient: authenticate success, global_id 4735 2024-03-25T18:36:02.942 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.940+0000 7f081a3ed640 10 monclient: _renew_subs 2024-03-25T18:36:02.942 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.940+0000 7f081a3ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:36:02.942 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.940+0000 7f081a3ed640 1 -- 172.21.15.155:0/2075847243 --> v1:172.21.15.167:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08141dfc30 con 0x7f0814061830 2024-03-25T18:36:02.942 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.940+0000 7f081a3ed640 10 client.4735.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:36:02.943 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.940+0000 7f081a3ed640 10 monclient: _renew_subs 2024-03-25T18:36:02.943 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.940+0000 7f081a3ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:36:02.943 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.940+0000 7f081a3ed640 1 -- 172.21.15.155:0/2075847243 --> v1:172.21.15.167:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f081413db40 con 0x7f0814061830 2024-03-25T18:36:02.943 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.940+0000 7f081a3ed640 1 librados: init done 2024-03-25T18:36:02.943 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.941+0000 7f081a3ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:02.943 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.941+0000 7f081a3ed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:02.943 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.941+0000 7f081a3ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:36:02.944 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.941+0000 7f081a3ed640 1 -- 172.21.15.155:0/2075847243 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08140627b0 con 0x7f0814061830 2024-03-25T18:36:02.965 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.964+0000 7f0810ff9640 1 -- 172.21.15.155:0/2075847243 <== mon.1 v1:172.21.15.167:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0804001cd0 con 0x7f0814061830 2024-03-25T18:36:02.965 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.964+0000 7f0810ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:02.965 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.964+0000 7f0810ff9640 1 -- 172.21.15.155:0/2075847243 <== mon.1 v1:172.21.15.167:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f0804003a60 con 0x7f0814061830 2024-03-25T18:36:02.965 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.964+0000 7f0810ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:02.965 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.964+0000 7f0810ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:36:02.965 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.964+0000 7f0810ff9640 10 monclient: dump: 2024-03-25T18:36:02.965 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:36:02.966 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:36:02.966 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:36:02.966 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:36:02.966 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:02.966 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:36:02.966 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:36:02.966 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:36:02.966 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:36:02.966 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:36:02.966 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.964+0000 7f08127fc640 4 set_mon_vals no callback set 2024-03-25T18:36:02.967 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.965+0000 7f0810ff9640 1 -- 172.21.15.155:0/2075847243 <== mon.1 v1:172.21.15.167:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7f0804003c80 con 0x7f0814061830 2024-03-25T18:36:02.967 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.966+0000 7f08137fe640 10 monclient: get_auth_request con 0x7f080006df70 auth_method 0 2024-03-25T18:36:02.967 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.966+0000 7f0810ff9640 10 client.4735.objecter ms_handle_connect 0x7f080006df70 2024-03-25T18:36:02.970 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.969+0000 7f0810ff9640 1 -- 172.21.15.155:0/2075847243 <== mon.1 v1:172.21.15.167:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1555029372 0 0) 0x7f0804054870 con 0x7f0814061830 2024-03-25T18:36:02.970 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.969+0000 7f0810ff9640 10 client.4735.objecter ms_dispatch 0x7f0814000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:36:02.970 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.969+0000 7f0810ff9640 3 client.4735.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:36:02.970 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.969+0000 7f0810ff9640 3 client.4735.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:36:02.970 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:02.969+0000 7f0810ff9640 20 client.4735.objecter dump_active .. 0 homeless 2024-03-25T18:36:03.235 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:03.234+0000 7f0810ff9640 1 -- 172.21.15.155:0/2075847243 <== mon.1 v1:172.21.15.167:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0804050380 con 0x7f0814061830 2024-03-25T18:36:03.235 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:03.234+0000 7f0810ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:03.235 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:03.234+0000 7f0810ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:36:03.449 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:03.448+0000 7f081a3ed640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:03.449 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:03.448+0000 7f081a3ed640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:03.449 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:03.448+0000 7f081a3ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:36:03.449 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:03.448+0000 7f081a3ed640 1 -- 172.21.15.155:0/2075847243 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0814069f40 con 0x7f0814061830 2024-03-25T18:36:03.472 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:03.471+0000 7f0810ff9640 1 -- 172.21.15.155:0/2075847243 <== mon.1 v1:172.21.15.167:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3170230695) 0x7f08040522b0 con 0x7f0814061830 2024-03-25T18:36:03.472 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:03.471+0000 7f0810ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:03.472 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:03.471+0000 7f0810ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:36:03.472 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:36:03.472 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":11,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:34:47.433406+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:34:47.433406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:36:03.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:03.473+0000 7f081a3ed640 10 librados: watch_flush enter 2024-03-25T18:36:03.475 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:03.473+0000 7f081a3ed640 10 librados: watch_flush exit 2024-03-25T18:36:03.475 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:03.473+0000 7f081a3ed640 20 client.4735.objecter shutdown clearing up homeless session... 2024-03-25T18:36:03.475 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:03.473+0000 7f081a3ed640 10 client.4735.objecter successfully canceled tick 2024-03-25T18:36:03.475 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:03.473+0000 7f081a3ed640 1 -- 172.21.15.155:0/2075847243 >> v1:172.21.15.167:6800/2919070503 conn(0x7f080006df70 legacy=0x7f0800070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:03.475 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:03.473+0000 7f081a3ed640 10 monclient: shutdown 2024-03-25T18:36:03.475 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:03.473+0000 7f081a3ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:03.475 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:03.473+0000 7f081a3ed640 1 -- 172.21.15.155:0/2075847243 >> v1:172.21.15.167:6789/0 conn(0x7f0814061830 legacy=0x7f08140aa240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:03.475 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:03.473+0000 7f081a3ed640 1 -- 172.21.15.155:0/2075847243 shutdown_connections 2024-03-25T18:36:03.475 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:03.473+0000 7f081a3ed640 1 -- 172.21.15.155:0/2075847243 >> 172.21.15.155:0/2075847243 conn(0x7f08140b11e0 msgr2=0x7f0814074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:03.475 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:03.473+0000 7f081a3ed640 1 -- 172.21.15.155:0/2075847243 shutdown_connections 2024-03-25T18:36:03.475 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:03.473+0000 7f081a3ed640 1 -- 172.21.15.155:0/2075847243 wait complete. 2024-03-25T18:36:03.475 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:03.473+0000 7f081a3ed640 1 librados: shutdown 2024-03-25T18:36:03.486 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:36:10.291 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:36:10.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.408+0000 7f61bbf3a640 10 monclient: get_monmap_and_config 2024-03-25T18:36:10.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.408+0000 7f61bbf3a640 10 monclient: build_initial_monmap 2024-03-25T18:36:10.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.408+0000 7f61bbf3a640 10 monclient: monmap: 2024-03-25T18:36:10.410 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:36:10.410 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:36:10.411 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:36:10.410367+0000 2024-03-25T18:36:10.411 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:36:10.410367+0000 2024-03-25T18:36:10.411 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:10.411 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:36:10.411 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:36:10.411 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-c 2024-03-25T18:36:10.411 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-25T18:36:10.411 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:36:10.412 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.409+0000 7f61bbf3a640 1 Processor -- start 2024-03-25T18:36:10.413 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.409+0000 7f61bbf3a640 1 -- start start 2024-03-25T18:36:10.413 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.409+0000 7f61bbf3a640 10 monclient: init 2024-03-25T18:36:10.413 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.410+0000 7f61bbf3a640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:10.413 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.410+0000 7f61bbf3a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:36:10.413 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.410+0000 7f61bbf3a640 10 monclient(hunting): picked mon.noname-a con 0x7f61b4149f50 addr v1:172.21.15.155:6789/0 2024-03-25T18:36:10.413 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.410+0000 7f61bbf3a640 10 monclient(hunting): picked mon.noname-b con 0x7f61b406c780 addr v1:172.21.15.155:6790/0 2024-03-25T18:36:10.413 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.410+0000 7f61bbf3a640 10 monclient(hunting): picked mon.noname-c con 0x7f61b40b1b10 addr v1:172.21.15.167:6789/0 2024-03-25T18:36:10.413 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.410+0000 7f61bbf3a640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61b414a330 con 0x7f61b4149f50 2024-03-25T18:36:10.413 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.410+0000 7f61bbf3a640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61b4070660 con 0x7f61b40b1b10 2024-03-25T18:36:10.414 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.410+0000 7f61bbf3a640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61b406d7c0 con 0x7f61b406c780 2024-03-25T18:36:10.414 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.410+0000 7f61bbf3a640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:10.414 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.410+0000 7f61bbf3a640 10 monclient(hunting): authenticate will time out at 8216.721390s 2024-03-25T18:36:10.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.426+0000 7f61b94d5640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f61b406c780 0x7f61b40b1400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:42872/0 (socket says 172.21.15.155:42872) 2024-03-25T18:36:10.428 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.426+0000 7f61b94d5640 1 -- 172.21.15.155:0/3835455437 learned_addr learned my addr 172.21.15.155:0/3835455437 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:36:10.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.438+0000 7f61b8cd4640 1 -- 172.21.15.155:0/3835455437 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2955611468 0 0) 0x7f61b406d7c0 con 0x7f61b406c780 2024-03-25T18:36:10.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.438+0000 7f61b8cd4640 10 monclient(hunting): _negotiate 2024-03-25T18:36:10.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.438+0000 7f61b8cd4640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:10.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.438+0000 7f61b8cd4640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:10.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.438+0000 7f61b8cd4640 10 monclient(hunting): my global_id is 4751 2024-03-25T18:36:10.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.438+0000 7f61b8cd4640 1 -- 172.21.15.155:0/3835455437 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61a0003520 con 0x7f61b406c780 2024-03-25T18:36:10.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.439+0000 7f61b8cd4640 1 -- 172.21.15.155:0/3835455437 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 548585486 0 0) 0x7f61b4070660 con 0x7f61b40b1b10 2024-03-25T18:36:10.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.439+0000 7f61b8cd4640 10 monclient(hunting): _negotiate 2024-03-25T18:36:10.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.439+0000 7f61b8cd4640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:10.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.439+0000 7f61b8cd4640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:10.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.439+0000 7f61b8cd4640 10 monclient(hunting): my global_id is 4738 2024-03-25T18:36:10.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.439+0000 7f61b8cd4640 1 -- 172.21.15.155:0/3835455437 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61b406d7c0 con 0x7f61b40b1b10 2024-03-25T18:36:10.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.443+0000 7f61b8cd4640 1 -- 172.21.15.155:0/3835455437 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 455469735 0 0) 0x7f61b414a330 con 0x7f61b4149f50 2024-03-25T18:36:10.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.443+0000 7f61b8cd4640 10 monclient(hunting): _negotiate 2024-03-25T18:36:10.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.443+0000 7f61b8cd4640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:10.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.443+0000 7f61b8cd4640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:10.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.443+0000 7f61b8cd4640 10 monclient(hunting): my global_id is 4740 2024-03-25T18:36:10.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.443+0000 7f61b8cd4640 1 -- 172.21.15.155:0/3835455437 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61b4070660 con 0x7f61b4149f50 2024-03-25T18:36:10.451 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.449+0000 7f61b8cd4640 1 -- 172.21.15.155:0/3835455437 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1548479025 0 0) 0x7f61b406d7c0 con 0x7f61b40b1b10 2024-03-25T18:36:10.451 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.449+0000 7f61b8cd4640 1 -- 172.21.15.155:0/3835455437 >> v1:172.21.15.155:6790/0 conn(0x7f61b406c780 legacy=0x7f61b40b1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:10.451 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.449+0000 7f61b8cd4640 1 -- 172.21.15.155:0/3835455437 >> v1:172.21.15.155:6789/0 conn(0x7f61b4149f50 legacy=0x7f61b40b0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:10.451 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.449+0000 7f61b8cd4640 10 monclient: _finish_hunting 0 2024-03-25T18:36:10.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.449+0000 7f61b8cd4640 1 monclient: found mon.noname-c 2024-03-25T18:36:10.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.449+0000 7f61b8cd4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:10.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.449+0000 7f61b8cd4640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.167:6789/0 2024-03-25T18:36:10.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.449+0000 7f61b8cd4640 1 -- 172.21.15.155:0/3835455437 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61b406d3e0 con 0x7f61b40b1b10 2024-03-25T18:36:10.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.449+0000 7f61b8cd4640 10 monclient: _finish_auth 0 2024-03-25T18:36:10.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.449+0000 7f61b8cd4640 10 monclient: _check_auth_tickets 2024-03-25T18:36:10.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.449+0000 7f61b8cd4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:10.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.449+0000 7f61b8cd4640 1 -- 172.21.15.155:0/3835455437 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f61ac0019f0 con 0x7f61b40b1b10 2024-03-25T18:36:10.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.449+0000 7f61bbf3a640 5 monclient: authenticate success, global_id 4738 2024-03-25T18:36:10.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.449+0000 7f61bbf3a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:36:10.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.449+0000 7f61b8cd4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:10.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.449+0000 7f61b8cd4640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:36:10.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.449+0000 7f61b8cd4640 10 monclient: dump: 2024-03-25T18:36:10.452 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:36:10.452 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:36:10.453 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:36:10.453 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:36:10.453 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:10.453 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:36:10.453 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:36:10.453 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:36:10.453 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:36:10.453 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:36:10.453 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.449+0000 7f61bbf3a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:36:10.460 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.458+0000 7f61b8cd4640 1 -- 172.21.15.155:0/3835455437 <== mon.1 v1:172.21.15.167:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f61ac001b50 con 0x7f61b40b1b10 2024-03-25T18:36:10.460 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.458+0000 7f61b8cd4640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:10.460 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.458+0000 7f61b8cd4640 1 -- 172.21.15.155:0/3835455437 <== mon.1 v1:172.21.15.167:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f61ac001f60 con 0x7f61b40b1b10 2024-03-25T18:36:10.460 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.458+0000 7f61b8cd4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:10.460 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.458+0000 7f61b8cd4640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:36:10.460 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.458+0000 7f61b8cd4640 10 monclient: dump: 2024-03-25T18:36:10.460 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:36:10.461 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:36:10.461 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:36:10.461 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:36:10.461 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:10.461 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:36:10.461 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:36:10.461 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:36:10.461 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:36:10.461 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:36:10.461 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.458+0000 7f61bbf3a640 10 monclient: get_monmap_and_config success 2024-03-25T18:36:10.461 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.458+0000 7f61bbf3a640 4 set_mon_vals no callback set 2024-03-25T18:36:10.463 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.459+0000 7f61bbf3a640 10 monclient: shutdown 2024-03-25T18:36:10.463 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.459+0000 7f61bbf3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:10.464 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.459+0000 7f61bbf3a640 1 -- 172.21.15.155:0/3835455437 >> v1:172.21.15.167:6789/0 conn(0x7f61b40b1b10 legacy=0x7f61b4152080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:10.464 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.459+0000 7f61bbf3a640 1 -- 172.21.15.155:0/3835455437 shutdown_connections 2024-03-25T18:36:10.464 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.459+0000 7f61bbf3a640 1 -- 172.21.15.155:0/3835455437 >> 172.21.15.155:0/3835455437 conn(0x7f61b40b5d10 msgr2=0x7f61b40b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:10.464 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.459+0000 7f61bbf3a640 1 -- 172.21.15.155:0/3835455437 shutdown_connections 2024-03-25T18:36:10.464 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.459+0000 7f61bbf3a640 1 -- 172.21.15.155:0/3835455437 wait complete. 2024-03-25T18:36:10.464 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.460+0000 7f61bbf3a640 10 monclient: build_initial_monmap 2024-03-25T18:36:10.464 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.460+0000 7f61bbf3a640 10 monclient: monmap: 2024-03-25T18:36:10.464 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-25T18:36:10.464 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:36:10.464 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-25T18:36:10.464 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-25T18:36:10.464 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:10.464 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-25T18:36:10.464 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-25T18:36:10.464 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.noname-b 2024-03-25T18:36:10.464 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-25T18:36:10.465 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:36:10.465 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.460+0000 7f61bbf3a640 1 librados: starting msgr at 2024-03-25T18:36:10.465 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.460+0000 7f61bbf3a640 1 librados: starting objecter 2024-03-25T18:36:10.465 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.460+0000 7f61bbf3a640 1 Processor -- start 2024-03-25T18:36:10.465 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.460+0000 7f61bbf3a640 1 -- start start 2024-03-25T18:36:10.465 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.460+0000 7f61bbf3a640 1 librados: setting wanted keys 2024-03-25T18:36:10.465 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.460+0000 7f61bbf3a640 1 librados: calling monclient init 2024-03-25T18:36:10.465 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.460+0000 7f61bbf3a640 10 monclient: init 2024-03-25T18:36:10.465 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.461+0000 7f61bbf3a640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:10.465 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.461+0000 7f61bbf3a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:36:10.465 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.461+0000 7f61bbf3a640 10 monclient(hunting): picked mon.noname-b con 0x7f61b406c780 addr v1:172.21.15.167:6789/0 2024-03-25T18:36:10.465 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.461+0000 7f61bbf3a640 10 monclient(hunting): picked mon.noname-c con 0x7f61b40b1b10 addr v1:172.21.15.155:6790/0 2024-03-25T18:36:10.465 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.461+0000 7f61bbf3a640 10 monclient(hunting): picked mon.noname-a con 0x7f61b4149f50 addr v1:172.21.15.155:6789/0 2024-03-25T18:36:10.465 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.461+0000 7f61bbf3a640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61b40aef70 con 0x7f61b4149f50 2024-03-25T18:36:10.465 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.461+0000 7f61bbf3a640 1 -- --> v1:172.21.15.167:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61b40af600 con 0x7f61b406c780 2024-03-25T18:36:10.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.461+0000 7f61bbf3a640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61b40af1e0 con 0x7f61b40b1b10 2024-03-25T18:36:10.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.461+0000 7f61bbf3a640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:10.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.461+0000 7f61bbf3a640 10 monclient(hunting): authenticate will time out at 8216.772228s 2024-03-25T18:36:10.468 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.466+0000 7f61b9cd6640 1 --1- >> v1:172.21.15.167:6789/0 conn(0x7f61b406c780 0x7f61b41dece0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.167:6789/0 says I am v1:172.21.15.155:36746/0 (socket says 172.21.15.155:36746) 2024-03-25T18:36:10.468 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.466+0000 7f61b9cd6640 1 -- 172.21.15.155:0/1563081032 learned_addr learned my addr 172.21.15.155:0/1563081032 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-25T18:36:10.469 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.467+0000 7f61aa7fc640 10 client.?.objecter ms_handle_connect 0x7f61b406c780 2024-03-25T18:36:10.469 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.467+0000 7f61aa7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:10.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.484+0000 7f61aa7fc640 1 -- 172.21.15.155:0/1563081032 <== mon.1 v1:172.21.15.167:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 21057117 0 0) 0x7f61b40af600 con 0x7f61b406c780 2024-03-25T18:36:10.487 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.484+0000 7f61aa7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:36:10.487 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.484+0000 7f61aa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:10.487 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.484+0000 7f61aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:10.487 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.484+0000 7f61aa7fc640 10 monclient(hunting): my global_id is 4741 2024-03-25T18:36:10.487 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.484+0000 7f61aa7fc640 1 -- 172.21.15.155:0/1563081032 --> v1:172.21.15.167:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6198004eb0 con 0x7f61b406c780 2024-03-25T18:36:10.487 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.485+0000 7f61aa7fc640 10 client.?.objecter ms_handle_connect 0x7f61b4149f50 2024-03-25T18:36:10.487 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.485+0000 7f61aa7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:10.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.489+0000 7f61aa7fc640 10 client.?.objecter ms_handle_connect 0x7f61b40b1b10 2024-03-25T18:36:10.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.489+0000 7f61aa7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:10.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.494+0000 7f61aa7fc640 1 -- 172.21.15.155:0/1563081032 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 150704139 0 0) 0x7f61b40aef70 con 0x7f61b4149f50 2024-03-25T18:36:10.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.494+0000 7f61aa7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:36:10.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.494+0000 7f61aa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:10.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.494+0000 7f61aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:10.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.494+0000 7f61aa7fc640 10 monclient(hunting): my global_id is 4743 2024-03-25T18:36:10.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.494+0000 7f61aa7fc640 1 -- 172.21.15.155:0/1563081032 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61b40af600 con 0x7f61b4149f50 2024-03-25T18:36:10.507 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.505+0000 7f61aa7fc640 1 -- 172.21.15.155:0/1563081032 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 862510109 0 0) 0x7f61b40af1e0 con 0x7f61b40b1b10 2024-03-25T18:36:10.507 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.505+0000 7f61aa7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:36:10.507 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.505+0000 7f61aa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:10.507 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.505+0000 7f61aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:10.507 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.505+0000 7f61aa7fc640 10 monclient(hunting): my global_id is 4754 2024-03-25T18:36:10.507 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.505+0000 7f61aa7fc640 1 -- 172.21.15.155:0/1563081032 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61b40aef70 con 0x7f61b40b1b10 2024-03-25T18:36:10.515 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.512+0000 7f61aa7fc640 1 -- 172.21.15.155:0/1563081032 <== mon.1 v1:172.21.15.167:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 456263314 0 0) 0x7f6198004eb0 con 0x7f61b406c780 2024-03-25T18:36:10.515 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.512+0000 7f61aa7fc640 1 -- 172.21.15.155:0/1563081032 --> v1:172.21.15.167:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f61b40af1e0 con 0x7f61b406c780 2024-03-25T18:36:10.515 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.512+0000 7f61aa7fc640 1 -- 172.21.15.155:0/1563081032 <== mon.1 v1:172.21.15.167:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f61b0001970 con 0x7f61b406c780 2024-03-25T18:36:10.515 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.512+0000 7f61aa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:10.515 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.512+0000 7f61aa7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:36:10.515 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.512+0000 7f61aa7fc640 10 monclient(hunting): dump: 2024-03-25T18:36:10.515 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:36:10.515 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:36:10.515 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:36:10.516 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:36:10.516 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:10.516 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:36:10.516 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:36:10.516 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:36:10.516 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:36:10.516 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:36:10.516 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.512+0000 7f61aa7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:36:10.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.521+0000 7f61aa7fc640 1 -- 172.21.15.155:0/1563081032 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1464996291 0 0) 0x7f61b40af600 con 0x7f61b4149f50 2024-03-25T18:36:10.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.521+0000 7f61aa7fc640 1 -- 172.21.15.155:0/1563081032 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6198004eb0 con 0x7f61b4149f50 2024-03-25T18:36:10.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.521+0000 7f61aa7fc640 1 -- 172.21.15.155:0/1563081032 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f61ac001e60 con 0x7f61b4149f50 2024-03-25T18:36:10.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.521+0000 7f61aa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:10.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.521+0000 7f61aa7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:36:10.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.521+0000 7f61aa7fc640 10 monclient(hunting): dump: 2024-03-25T18:36:10.524 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:36:10.524 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:36:10.524 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:36:10.524 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:36:10.524 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:10.524 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:36:10.524 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:36:10.524 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:36:10.524 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:36:10.525 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:36:10.531 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.528+0000 7f61aa7fc640 1 -- 172.21.15.155:0/1563081032 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 923987231 0 0) 0x7f61b40aef70 con 0x7f61b40b1b10 2024-03-25T18:36:10.531 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.528+0000 7f61aa7fc640 1 -- 172.21.15.155:0/1563081032 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f61b40af600 con 0x7f61b40b1b10 2024-03-25T18:36:10.531 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.528+0000 7f61aa7fc640 1 -- 172.21.15.155:0/1563081032 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f61a40017b0 con 0x7f61b40b1b10 2024-03-25T18:36:10.531 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.529+0000 7f61aa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:10.531 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.529+0000 7f61aa7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:36:10.531 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.529+0000 7f61aa7fc640 10 monclient(hunting): dump: 2024-03-25T18:36:10.531 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:36:10.531 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:36:10.531 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:36:10.531 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:36:10.531 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:10.531 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:36:10.531 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:36:10.532 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:36:10.532 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:36:10.532 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:36:10.538 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.535+0000 7f61aa7fc640 1 -- 172.21.15.155:0/1563081032 <== mon.1 v1:172.21.15.167:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3916485289 0 0) 0x7f61b40af1e0 con 0x7f61b406c780 2024-03-25T18:36:10.538 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.535+0000 7f61aa7fc640 1 -- 172.21.15.155:0/1563081032 >> v1:172.21.15.155:6790/0 conn(0x7f61b40b1b10 legacy=0x7f61b40afdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:10.538 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.535+0000 7f61aa7fc640 1 -- 172.21.15.155:0/1563081032 >> v1:172.21.15.155:6789/0 conn(0x7f61b4149f50 legacy=0x7f61b40acb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:10.538 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.535+0000 7f61aa7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:36:10.538 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.535+0000 7f61aa7fc640 1 monclient: found mon.b 2024-03-25T18:36:10.538 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.535+0000 7f61aa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:10.538 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.535+0000 7f61aa7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:36:10.538 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.535+0000 7f61aa7fc640 1 -- 172.21.15.155:0/1563081032 --> v1:172.21.15.167:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61b40af8a0 con 0x7f61b406c780 2024-03-25T18:36:10.538 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.535+0000 7f61aa7fc640 10 monclient: _finish_auth 0 2024-03-25T18:36:10.538 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.535+0000 7f61aa7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:36:10.538 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.535+0000 7f61aa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:10.538 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.535+0000 7f61bbf3a640 5 monclient: authenticate success, global_id 4741 2024-03-25T18:36:10.539 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.535+0000 7f61bbf3a640 10 monclient: _renew_subs 2024-03-25T18:36:10.539 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.535+0000 7f61bbf3a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:36:10.539 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.535+0000 7f61bbf3a640 1 -- 172.21.15.155:0/1563081032 --> v1:172.21.15.167:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61b40adba0 con 0x7f61b406c780 2024-03-25T18:36:10.539 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.535+0000 7f61bbf3a640 10 client.4741.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:36:10.539 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.535+0000 7f61bbf3a640 10 monclient: _renew_subs 2024-03-25T18:36:10.539 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.535+0000 7f61bbf3a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:36:10.539 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.535+0000 7f61bbf3a640 1 -- 172.21.15.155:0/1563081032 --> v1:172.21.15.167:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f61b40ae030 con 0x7f61b406c780 2024-03-25T18:36:10.539 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.535+0000 7f61bbf3a640 1 librados: init done 2024-03-25T18:36:10.540 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.536+0000 7f61bbf3a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:10.540 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.536+0000 7f61bbf3a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:10.540 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.537+0000 7f61bbf3a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:36:10.540 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.537+0000 7f61bbf3a640 1 -- 172.21.15.155:0/1563081032 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61b40b15e0 con 0x7f61b406c780 2024-03-25T18:36:10.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.850+0000 7f61aa7fc640 1 -- 172.21.15.155:0/1563081032 <== mon.1 v1:172.21.15.167:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f61b0001ad0 con 0x7f61b406c780 2024-03-25T18:36:10.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.850+0000 7f61aa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:10.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.850+0000 7f61aa7fc640 1 -- 172.21.15.155:0/1563081032 <== mon.1 v1:172.21.15.167:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3512391374 0 0) 0x7f61b0003a60 con 0x7f61b406c780 2024-03-25T18:36:10.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.850+0000 7f61aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:10.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.850+0000 7f61aa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:36:10.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.850+0000 7f61aa7fc640 10 monclient: dump: 2024-03-25T18:36:10.852 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-25T18:36:10.852 INFO:teuthology.misc.health.smithi155.stderr:fsid 80c12a94-0fcf-4c12-99dd-38ab8a9afe2b 2024-03-25T18:36:10.852 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-25T18:23:53.480304+0000 2024-03-25T18:36:10.852 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-25T18:23:53.480304+0000 2024-03-25T18:36:10.852 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:10.852 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 3 2024-03-25T18:36:10.853 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-25T18:36:10.853 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.167:6789/0 mon.b 2024-03-25T18:36:10.853 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-25T18:36:10.853 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-25T18:36:10.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.850+0000 7f61abfff640 4 set_mon_vals no callback set 2024-03-25T18:36:10.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.851+0000 7f61aa7fc640 1 -- 172.21.15.155:0/1563081032 <== mon.1 v1:172.21.15.167:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2738529973 0 0) 0x7f61b0003c40 con 0x7f61b406c780 2024-03-25T18:36:10.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.851+0000 7f61b94d5640 10 monclient: get_auth_request con 0x7f619806e0f0 auth_method 0 2024-03-25T18:36:10.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.851+0000 7f61aa7fc640 10 client.4741.objecter ms_handle_connect 0x7f619806e0f0 2024-03-25T18:36:10.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.858+0000 7f61aa7fc640 1 -- 172.21.15.155:0/1563081032 <== mon.1 v1:172.21.15.167:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1555029372 0 0) 0x7f61b0054850 con 0x7f61b406c780 2024-03-25T18:36:10.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.858+0000 7f61aa7fc640 10 client.4741.objecter ms_dispatch 0x7f61b4000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:36:10.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.858+0000 7f61aa7fc640 3 client.4741.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:36:10.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.858+0000 7f61aa7fc640 3 client.4741.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:36:10.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.858+0000 7f61aa7fc640 20 client.4741.objecter dump_active .. 0 homeless 2024-03-25T18:36:10.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.858+0000 7f61aa7fc640 1 -- 172.21.15.155:0/1563081032 <== mon.1 v1:172.21.15.167:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f61b0050360 con 0x7f61b406c780 2024-03-25T18:36:10.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.858+0000 7f61aa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:10.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:10.858+0000 7f61aa7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:36:11.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:11.102+0000 7f61bbf3a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:11.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:11.102+0000 7f61bbf3a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:11.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:11.102+0000 7f61bbf3a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.167:6789/0 2024-03-25T18:36:11.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:11.102+0000 7f61bbf3a640 1 -- 172.21.15.155:0/1563081032 --> v1:172.21.15.167:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f61b40b4d40 con 0x7f61b406c780 2024-03-25T18:36:11.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:11.137+0000 7f61aa7fc640 1 -- 172.21.15.155:0/1563081032 <== mon.1 v1:172.21.15.167:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3170230695) 0x7f61b0052290 con 0x7f61b406c780 2024-03-25T18:36:11.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:11.137+0000 7f61aa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:11.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:11.137+0000 7f61aa7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:36:11.139 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-25T18:36:11.139 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":11,"fsid":"80c12a94-0fcf-4c12-99dd-38ab8a9afe2b","created":"2024-03-25T18:24:43.239998+0000","modified":"2024-03-25T18:34:47.433406+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:34:47.433406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"359f8cfa-5c58-4b11-ad25-4fac6f4d43ba","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"d22b6af2-efd4-4433-a24d-1117b2c8681d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"03747c34-7e23-428a-8e35-b6132f7e8ae7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"ba181ae1-e467-47ee-b719-810d7d0b9e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5964583a-b9a0-424e-b387-589aa8f8b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bfddfbda-7f87-471d-a105-bc2766a64f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4959b0-29a4-4797-8a9a-af1749e2524f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5c3d7044-9a4b-42ee-941e-90eb460fd371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:36:11.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:11.139+0000 7f61bbf3a640 10 librados: watch_flush enter 2024-03-25T18:36:11.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:11.139+0000 7f61bbf3a640 10 librados: watch_flush exit 2024-03-25T18:36:11.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:11.139+0000 7f61bbf3a640 20 client.4741.objecter shutdown clearing up homeless session... 2024-03-25T18:36:11.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:11.139+0000 7f61bbf3a640 10 client.4741.objecter successfully canceled tick 2024-03-25T18:36:11.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:11.139+0000 7f61bbf3a640 1 -- 172.21.15.155:0/1563081032 >> v1:172.21.15.167:6800/2919070503 conn(0x7f619806e0f0 legacy=0x7f61980705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:11.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:11.139+0000 7f61bbf3a640 10 monclient: shutdown 2024-03-25T18:36:11.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:11.139+0000 7f61bbf3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:11.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:11.139+0000 7f61bbf3a640 1 -- 172.21.15.155:0/1563081032 >> v1:172.21.15.167:6789/0 conn(0x7f61b406c780 legacy=0x7f61b41dece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:11.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:11.139+0000 7f61bbf3a640 1 -- 172.21.15.155:0/1563081032 shutdown_connections 2024-03-25T18:36:11.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:11.139+0000 7f61bbf3a640 1 -- 172.21.15.155:0/1563081032 >> 172.21.15.155:0/1563081032 conn(0x7f61b40b5d10 msgr2=0x7f61b4143bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:11.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:11.139+0000 7f61bbf3a640 1 -- 172.21.15.155:0/1563081032 shutdown_connections 2024-03-25T18:36:11.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:11.139+0000 7f61bbf3a640 1 -- 172.21.15.155:0/1563081032 wait complete. 2024-03-25T18:36:11.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-25T18:36:11.140+0000 7f61bbf3a640 1 librados: shutdown 2024-03-25T18:36:11.155 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:36:11.155 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-25T18:36:11.156 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-25T18:36:11.157 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-25T18:36:11.157 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:36:12.888 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:11:46.493 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:12.892 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:11:46.471 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:12.906 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:11:46.485 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:12.907 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:11:46.512 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:12.938 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:11:46.517 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:12.943 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:11:46.548 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:12.958 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:11:46.563 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:12.964 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:11:46.569 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:12.992 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:11:46.596 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:13.213 INFO:tasks.ceph.osd.0.smithi155.stderr:==00:00:11:24.585 35524== 2024-03-25T18:36:13.214 INFO:tasks.ceph.osd.0.smithi155.stderr:==00:00:11:24.585 35524== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:36:13.256 INFO:tasks.ceph.osd.0.smithi155.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:36:17.258 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:36:17.259 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi155 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0" 2024-03-25T18:36:17.259 INFO:tasks.ceph.osd.0:Stopped 2024-03-25T18:36:17.260 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-25T18:36:17.260 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:36:19.330 INFO:tasks.ceph.osd.1.smithi155.stderr:==00:00:11:30.695 35525== 2024-03-25T18:36:19.330 INFO:tasks.ceph.osd.1.smithi155.stderr:==00:00:11:30.695 35525== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:36:19.368 INFO:tasks.ceph.osd.1.smithi155.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:36:23.361 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:36:23.361 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi155 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1" 2024-03-25T18:36:23.362 INFO:tasks.ceph.osd.1:Stopped 2024-03-25T18:36:23.362 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-25T18:36:23.362 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:36:25.219 INFO:tasks.ceph.osd.2.smithi155.stderr:==00:00:11:36.601 35523== 2024-03-25T18:36:25.219 INFO:tasks.ceph.osd.2.smithi155.stderr:==00:00:11:36.601 35523== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:36:25.256 INFO:tasks.ceph.osd.2.smithi155.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:36:29.464 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:36:29.464 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi155 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2" 2024-03-25T18:36:29.465 INFO:tasks.ceph.osd.2:Stopped 2024-03-25T18:36:29.465 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-25T18:36:29.465 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:36:31.391 INFO:tasks.ceph.osd.3.smithi155.stderr:==00:00:11:42.750 35526== 2024-03-25T18:36:31.391 INFO:tasks.ceph.osd.3.smithi155.stderr:==00:00:11:42.750 35526== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:36:31.429 INFO:tasks.ceph.osd.3.smithi155.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:36:32.918 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:12:06.522 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:32.924 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:12:06.529 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:32.945 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:12:06.550 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:32.961 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:12:06.566 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:32.971 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:12:06.576 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:32.975 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:12:06.554 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:32.981 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:12:06.559 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:32.993 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:12:06.598 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:33.002 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:12:06.581 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:35.566 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:36:35.567 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi155 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3" 2024-03-25T18:36:35.567 INFO:tasks.ceph.osd.3:Stopped 2024-03-25T18:36:35.567 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-25T18:36:35.567 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:36:37.469 INFO:tasks.ceph.osd.4.smithi167.stderr:==00:00:11:48.823 35187== 2024-03-25T18:36:37.469 INFO:tasks.ceph.osd.4.smithi167.stderr:==00:00:11:48.823 35187== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:36:37.510 INFO:tasks.ceph.osd.4.smithi167.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:36:41.669 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:36:41.669 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4" 2024-03-25T18:36:41.670 INFO:tasks.ceph.osd.4:Stopped 2024-03-25T18:36:41.670 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-25T18:36:41.670 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:36:43.644 INFO:tasks.ceph.osd.5.smithi167.stderr:==00:00:11:55.016 35184== 2024-03-25T18:36:43.644 INFO:tasks.ceph.osd.5.smithi167.stderr:==00:00:11:55.016 35184== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:36:43.679 INFO:tasks.ceph.osd.5.smithi167.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:36:47.772 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:36:47.773 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5" 2024-03-25T18:36:47.773 INFO:tasks.ceph.osd.5:Stopped 2024-03-25T18:36:47.773 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-25T18:36:47.773 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:36:49.582 INFO:tasks.ceph.osd.6.smithi167.stderr:==00:00:12:00.952 35185== 2024-03-25T18:36:49.582 INFO:tasks.ceph.osd.6.smithi167.stderr:==00:00:12:00.952 35185== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:36:49.623 INFO:tasks.ceph.osd.6.smithi167.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:36:52.930 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:12:26.535 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:52.975 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:12:26.580 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:52.976 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:12:26.555 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:52.991 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:12:26.596 35208== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:53.022 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:12:26.601 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:53.023 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:12:26.627 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:53.028 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:12:26.633 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:53.041 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:12:26.620 34641== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:53.048 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:12:26.653 35211== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:53.875 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:36:53.875 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6" 2024-03-25T18:36:53.876 INFO:tasks.ceph.osd.6:Stopped 2024-03-25T18:36:53.876 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-25T18:36:53.876 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:36:55.328 INFO:tasks.ceph.osd.7.smithi167.stderr:==00:00:12:06.695 35186== 2024-03-25T18:36:55.329 INFO:tasks.ceph.osd.7.smithi167.stderr:==00:00:12:06.695 35186== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:36:55.358 INFO:tasks.ceph.osd.7.smithi167.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:36:59.978 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:36:59.978 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7" 2024-03-25T18:36:59.979 INFO:tasks.ceph.osd.7:Stopped 2024-03-25T18:36:59.979 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-25T18:36:59.979 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-25T18:36:59.979 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:37:00.007 INFO:tasks.ceph.mgr.y:Stopped 2024-03-25T18:37:00.007 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-25T18:37:00.007 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:37:00.008 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-03-25T18:37:00.007+0000 7f181bee9640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i x (PID: 34639) UID: 0 2024-03-25T18:37:00.036 INFO:tasks.ceph.mgr.x:Stopped 2024-03-25T18:37:00.036 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-25T18:37:00.036 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-25T18:37:00.037 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:37:00.062 INFO:tasks.ceph.mon.a.smithi155.stderr:2024-03-25T18:37:00.050+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 35177) UID: 0 2024-03-25T18:37:00.063 INFO:tasks.ceph.mon.a.smithi155.stderr:2024-03-25T18:37:00.052+0000 9f86640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-25T18:37:00.468 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:12:34.073 35208== 2024-03-25T18:37:00.468 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:12:34.073 35208== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:37:00.477 INFO:tasks.ceph.mon.a.smithi155.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:37:06.142 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:37:06.273 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi155 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-03-25T18:37:06.274 INFO:tasks.ceph.mon.a:Stopped 2024-03-25T18:37:06.274 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-25T18:37:06.274 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:37:06.290 INFO:tasks.ceph.mon.c.smithi155.stderr:2024-03-25T18:37:06.278+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 35179) UID: 0 2024-03-25T18:37:06.290 INFO:tasks.ceph.mon.c.smithi155.stderr:2024-03-25T18:37:06.279+0000 9f86640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-03-25T18:37:06.918 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:12:40.523 35211== 2024-03-25T18:37:06.918 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:12:40.523 35211== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:37:06.927 INFO:tasks.ceph.mon.c.smithi155.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:37:12.377 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:37:12.377 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi155 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-03-25T18:37:12.378 INFO:tasks.ceph.mon.c:Stopped 2024-03-25T18:37:12.378 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-25T18:37:12.378 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:37:12.394 INFO:tasks.ceph.mon.b.smithi167.stderr:2024-03-25T18:37:12.382+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 34637) UID: 0 2024-03-25T18:37:12.394 INFO:tasks.ceph.mon.b.smithi167.stderr:2024-03-25T18:37:12.383+0000 9f86640 -1 mon.b@1(probing) e1 *** Got Signal Terminated *** 2024-03-25T18:37:12.831 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:12:46.409 34641== 2024-03-25T18:37:12.831 INFO:tasks.ceph.mon.b.smithi167.stderr:==00:00:12:46.409 34641== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:37:12.840 INFO:tasks.ceph.mon.b.smithi167.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:37:18.479 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:37:18.479 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-03-25T18:37:18.480 INFO:tasks.ceph.mon.b:Stopped 2024-03-25T18:37:18.480 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-25T18:37:18.480 DEBUG:teuthology.orchestra.run.smithi155:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-03-25T18:37:18.592 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi155.front.sepia.ceph.com 2024-03-25T18:37:18.592 DEBUG:teuthology.orchestra.run.smithi155:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-25T18:37:18.697 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi155.front.sepia.ceph.com 2024-03-25T18:37:18.697 DEBUG:teuthology.orchestra.run.smithi155:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-25T18:37:18.762 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi155.front.sepia.ceph.com 2024-03-25T18:37:18.762 DEBUG:teuthology.orchestra.run.smithi155:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-25T18:37:18.839 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi155.front.sepia.ceph.com 2024-03-25T18:37:18.839 DEBUG:teuthology.orchestra.run.smithi155:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-25T18:37:18.928 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi167.front.sepia.ceph.com 2024-03-25T18:37:18.928 DEBUG:teuthology.orchestra.run.smithi167:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-25T18:37:19.055 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi167.front.sepia.ceph.com 2024-03-25T18:37:19.056 DEBUG:teuthology.orchestra.run.smithi167:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-25T18:37:19.127 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi167.front.sepia.ceph.com 2024-03-25T18:37:19.127 DEBUG:teuthology.orchestra.run.smithi167:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-25T18:37:19.193 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi167.front.sepia.ceph.com 2024-03-25T18:37:19.193 DEBUG:teuthology.orchestra.run.smithi167:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-25T18:37:19.258 INFO:tasks.ceph:Archiving mon data... 2024-03-25T18:37:19.260 DEBUG:teuthology.misc:Transferring archived files from smithi155:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621273/data/mon.a.tgz 2024-03-25T18:37:19.260 DEBUG:teuthology.orchestra.run.smithi155:> mktemp 2024-03-25T18:37:19.275 INFO:teuthology.orchestra.run.smithi155.stdout:/tmp/tmp.4gnPjLZm28 2024-03-25T18:37:19.276 DEBUG:teuthology.orchestra.run.smithi155:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.4gnPjLZm28 2024-03-25T18:37:19.383 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 0666 /tmp/tmp.4gnPjLZm28 2024-03-25T18:37:19.468 DEBUG:teuthology.orchestra.remote:smithi155:/tmp/tmp.4gnPjLZm28 is 132KB 2024-03-25T18:37:19.520 DEBUG:teuthology.orchestra.run.smithi155:> rm -fr /tmp/tmp.4gnPjLZm28 2024-03-25T18:37:19.534 DEBUG:teuthology.misc:Transferring archived files from smithi155:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621273/data/mon.c.tgz 2024-03-25T18:37:19.535 DEBUG:teuthology.orchestra.run.smithi155:> mktemp 2024-03-25T18:37:19.589 INFO:teuthology.orchestra.run.smithi155.stdout:/tmp/tmp.wQyRTTtukC 2024-03-25T18:37:19.589 DEBUG:teuthology.orchestra.run.smithi155:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.wQyRTTtukC 2024-03-25T18:37:19.678 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 0666 /tmp/tmp.wQyRTTtukC 2024-03-25T18:37:19.763 DEBUG:teuthology.orchestra.remote:smithi155:/tmp/tmp.wQyRTTtukC is 133KB 2024-03-25T18:37:19.814 DEBUG:teuthology.orchestra.run.smithi155:> rm -fr /tmp/tmp.wQyRTTtukC 2024-03-25T18:37:19.828 DEBUG:teuthology.misc:Transferring archived files from smithi167:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621273/data/mon.b.tgz 2024-03-25T18:37:19.828 DEBUG:teuthology.orchestra.run.smithi167:> mktemp 2024-03-25T18:37:19.845 INFO:teuthology.orchestra.run.smithi167.stdout:/tmp/tmp.n2jtBaJQdV 2024-03-25T18:37:19.846 DEBUG:teuthology.orchestra.run.smithi167:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.n2jtBaJQdV 2024-03-25T18:37:19.950 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod 0666 /tmp/tmp.n2jtBaJQdV 2024-03-25T18:37:20.037 DEBUG:teuthology.orchestra.remote:smithi167:/tmp/tmp.n2jtBaJQdV is 133KB 2024-03-25T18:37:20.089 DEBUG:teuthology.orchestra.run.smithi167:> rm -fr /tmp/tmp.n2jtBaJQdV 2024-03-25T18:37:20.103 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-25T18:37:20.103 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T18:37:20.106 DEBUG:teuthology.orchestra.run.smithi167:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T18:37:20.488 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:37:20.679 INFO:tasks.ceph:Archiving crash dumps... 2024-03-25T18:37:20.680 DEBUG:teuthology.misc:Transferring archived files from smithi155:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621273/remote/smithi155/crash 2024-03-25T18:37:20.681 DEBUG:teuthology.orchestra.run.smithi155:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T18:37:20.708 DEBUG:teuthology.misc:Transferring archived files from smithi167:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621273/remote/smithi167/crash 2024-03-25T18:37:20.709 DEBUG:teuthology.orchestra.run.smithi167:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T18:37:20.737 INFO:tasks.ceph:Compressing logs... 2024-03-25T18:37:20.737 DEBUG:teuthology.orchestra.run.smithi155:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T18:37:20.753 DEBUG:teuthology.orchestra.run.smithi167:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T18:37:20.826 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-25T18:37:20.826 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34994.log 2024-03-25T18:37:20.826 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34954.log 2024-03-25T18:37:20.826 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.34994.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2024-03-25T18:37:20.826 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.34954.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-03-25T18:37:20.826 INFO:teuthology.orchestra.run.smithi167.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34994.log.gz 2024-03-25T18:37:20.826 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5/var/log/ceph/valgrind/osd.7.log: --verbose -- /var/log/ceph/valgrind/osd.6.log 2024-03-25T18:37:20.826 INFO:teuthology.orchestra.run.smithi167.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34954.log.gz 2024-03-25T18:37:20.826 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/valgrind/osd.5.log: gzip -5 --verbose 72.1% -- -- replaced with /var/log/ceph/valgrind/osd.7.log.gz /var/log/ceph/valgrind/mon.b.log 2024-03-25T18:37:20.827 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:37:20.827 INFO:teuthology.orchestra.run.smithi167.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-03-25T18:37:20.827 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/valgrind/osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-03-25T18:37:20.827 INFO:teuthology.orchestra.run.smithi167.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-03-25T18:37:20.827 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T18:37:20.827 INFO:teuthology.orchestra.run.smithi167.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-03-25T18:37:20.827 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/valgrind/osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-25T18:37:20.827 INFO:teuthology.orchestra.run.smithi167.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-03-25T18:37:20.827 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34834.log 2024-03-25T18:37:20.827 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34874.log 2024-03-25T18:37:20.827 INFO:teuthology.orchestra.run.smithi167.stderr: 88.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T18:37:20.827 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.34834.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-25T18:37:20.827 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.34874.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.34834.log.gz 2024-03-25T18:37:20.827 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-25T18:37:20.828 INFO:teuthology.orchestra.run.smithi167.stderr:gzip 87.5% -5 -- replaced with /var/log/ceph/ceph-client.admin.34874.log.gz 2024-03-25T18:37:20.828 INFO:teuthology.orchestra.run.smithi167.stderr: --verbose -- /var/log/ceph/ceph-client.admin.35085.log 2024-03-25T18:37:20.828 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-osd.7.log: /var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34914.log 2024-03-25T18:37:20.828 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.35085.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-25T18:37:20.828 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.34914.log: gzip -5 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35085.log.gz 2024-03-25T18:37:20.828 INFO:teuthology.orchestra.run.smithi167.stderr: --verbose -- /var/log/ceph/ceph-client.admin.35036.log 2024-03-25T18:37:20.828 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 87.7% --verbose -- /var/log/ceph/ceph.log 2024-03-25T18:37:20.828 INFO:teuthology.orchestra.run.smithi167.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34914.log.gz 2024-03-25T18:37:20.828 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.35036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34794.log 2024-03-25T18:37:20.828 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph.log: 87.6%gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-25T18:37:20.828 INFO:teuthology.orchestra.run.smithi167.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35036.log.gz 2024-03-25T18:37:20.828 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.34794.log: /var/log/ceph/ceph-mon.b.log: 87.4% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T18:37:20.828 INFO:teuthology.orchestra.run.smithi167.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.34794.log.gz 2024-03-25T18:37:20.834 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40707.log 2024-03-25T18:37:20.834 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41956.log 2024-03-25T18:37:20.834 INFO:teuthology.orchestra.run.smithi155.stderr:gzip/var/log/ceph/ceph-client.admin.40707.log: -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-25T18:37:20.834 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41293.log 2024-03-25T18:37:20.834 INFO:teuthology.orchestra.run.smithi155.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40707.log.gz 2024-03-25T18:37:20.834 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39731.log 2024-03-25T18:37:20.834 INFO:teuthology.orchestra.run.smithi155.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41956.log.gz 2024-03-25T18:37:20.835 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39496.log 2024-03-25T18:37:20.835 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41293.log: /var/log/ceph/ceph-client.admin.39731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41410.log 2024-03-25T18:37:20.835 INFO:teuthology.orchestra.run.smithi155.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41293.log.gz 2024-03-25T18:37:20.835 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39301.log 2024-03-25T18:37:20.835 INFO:teuthology.orchestra.run.smithi155.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39731.log.gz 2024-03-25T18:37:20.835 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40394.log 2024-03-25T18:37:20.835 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41410.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39496.log.gz 2024-03-25T18:37:20.835 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41254.log 2024-03-25T18:37:20.835 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40394.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41410.log.gz 2024-03-25T18:37:20.835 INFO:teuthology.orchestra.run.smithi155.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39301.log.gz 2024-03-25T18:37:20.835 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-03-25T18:37:20.835 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41254.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-03-25T18:37:20.835 INFO:teuthology.orchestra.run.smithi155.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40394.log.gz 2024-03-25T18:37:20.835 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/valgrind/mon.c.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2024-03-25T18:37:20.836 INFO:teuthology.orchestra.run.smithi155.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41254.log.gz 2024-03-25T18:37:20.836 INFO:teuthology.orchestra.run.smithi155.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-03-25T18:37:20.836 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/valgrind/osd.2.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-03-25T18:37:20.836 INFO:teuthology.orchestra.run.smithi155.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-03-25T18:37:20.836 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/valgrind/osd.0.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-03-25T18:37:20.836 INFO:teuthology.orchestra.run.smithi155.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-03-25T18:37:20.836 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/valgrind/osd.1.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-03-25T18:37:20.836 INFO:teuthology.orchestra.run.smithi155.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-03-25T18:37:20.836 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/valgrind/mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40512.log 2024-03-25T18:37:20.836 INFO:teuthology.orchestra.run.smithi155.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-03-25T18:37:20.836 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/valgrind/osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42151.log 2024-03-25T18:37:20.836 INFO:teuthology.orchestra.run.smithi155.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-03-25T18:37:20.836 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41371.log 2024-03-25T18:37:20.836 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.42151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40004.log 2024-03-25T18:37:20.836 INFO:teuthology.orchestra.run.smithi155.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40512.log.gz 2024-03-25T18:37:20.837 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40864.log 2024-03-25T18:37:20.837 INFO:teuthology.orchestra.run.smithi155.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.42151.log.gz 2024-03-25T18:37:20.837 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39535.log 2024-03-25T18:37:20.837 INFO:teuthology.orchestra.run.smithi155.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41371.log.gz 2024-03-25T18:37:20.837 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40864.log: gzip -5 86.5% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.40004.log.gz 2024-03-25T18:37:20.837 INFO:teuthology.orchestra.run.smithi155.stderr: /var/log/ceph/ceph-client.admin.40785.log 2024-03-25T18:37:20.837 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41800.log 2024-03-25T18:37:20.837 INFO:teuthology.orchestra.run.smithi155.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40864.log.gz 2024-03-25T18:37:20.837 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39965.log 2024-03-25T18:37:20.837 INFO:teuthology.orchestra.run.smithi155.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39535.log.gz 2024-03-25T18:37:20.837 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41800.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40785.log.gz 2024-03-25T18:37:20.837 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39809.log 2024-03-25T18:37:20.837 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42346.log 2024-03-25T18:37:20.837 INFO:teuthology.orchestra.run.smithi155.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41800.log.gz 2024-03-25T18:37:20.838 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39064.log 2024-03-25T18:37:20.838 INFO:teuthology.orchestra.run.smithi155.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39965.log.gz 2024-03-25T18:37:20.838 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.42346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41839.log 2024-03-25T18:37:20.838 INFO:teuthology.orchestra.run.smithi155.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39809.log.gz 2024-03-25T18:37:20.838 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40472.log 2024-03-25T18:37:20.838 INFO:teuthology.orchestra.run.smithi155.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.42346.log.gz 2024-03-25T18:37:20.838 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40199.log 2024-03-25T18:37:20.838 INFO:teuthology.orchestra.run.smithi155.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39064.log.gz 2024-03-25T18:37:20.838 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40472.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41839.log.gz 2024-03-25T18:37:20.838 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41332.log 2024-03-25T18:37:20.838 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42307.log 2024-03-25T18:37:20.838 INFO:teuthology.orchestra.run.smithi155.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40472.log.gz 2024-03-25T18:37:20.838 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41878.log 2024-03-25T18:37:20.838 INFO:teuthology.orchestra.run.smithi155.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40199.log.gz 2024-03-25T18:37:20.839 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.42307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39652.log 2024-03-25T18:37:20.839 INFO:teuthology.orchestra.run.smithi155.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41332.log.gz 2024-03-25T18:37:20.839 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39848.log 2024-03-25T18:37:20.839 INFO:teuthology.orchestra.run.smithi155.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.42307.log.gz 2024-03-25T18:37:20.839 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39652.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T18:37:20.839 INFO:teuthology.orchestra.run.smithi155.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41878.log.gz 2024-03-25T18:37:20.839 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39848.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39652.log.gz 2024-03-25T18:37:20.839 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35683.log 2024-03-25T18:37:20.839 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40316.log 2024-03-25T18:37:20.839 INFO:teuthology.orchestra.run.smithi155.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39848.log.gz 2024-03-25T18:37:20.839 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.35683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39262.log 2024-03-25T18:37:20.839 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40316.log: 93.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T18:37:20.839 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39145.log 2024-03-25T18:37:20.839 INFO:teuthology.orchestra.run.smithi155.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35683.log.gz 2024-03-25T18:37:20.839 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35549.log 2024-03-25T18:37:20.840 INFO:teuthology.orchestra.run.smithi155.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40316.log.gz 2024-03-25T18:37:20.840 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39574.log 2024-03-25T18:37:20.840 INFO:teuthology.orchestra.run.smithi155.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39262.log.gz 2024-03-25T18:37:20.840 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.35549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41683.log 86.7% 2024-03-25T18:37:20.840 INFO:teuthology.orchestra.run.smithi155.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39145.log.gz 2024-03-25T18:37:20.840 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39574.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.35549.log.gz 2024-03-25T18:37:20.840 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41527.log 2024-03-25T18:37:20.840 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41683.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39574.log.gz 2024-03-25T18:37:20.840 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40981.log 2024-03-25T18:37:20.840 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41527.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-25T18:37:20.840 INFO:teuthology.orchestra.run.smithi155.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41683.log.gz 2024-03-25T18:37:20.840 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40981.log: gzip -5 86.6% --verbose -- replaced with /var/log/ceph/ceph-client.admin.41527.log.gz -- 2024-03-25T18:37:20.840 INFO:teuthology.orchestra.run.smithi155.stderr: /var/log/ceph/ceph-client.admin.41917.log 2024-03-25T18:37:20.840 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40160.log 2024-03-25T18:37:20.841 INFO:teuthology.orchestra.run.smithi155.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40981.log.gz 2024-03-25T18:37:20.841 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40629.log 2024-03-25T18:37:20.841 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35219.log 2024-03-25T18:37:20.841 INFO:teuthology.orchestra.run.smithi155.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41917.log.gz 2024-03-25T18:37:20.841 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40629.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40160.log.gz 2024-03-25T18:37:20.841 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41761.log 2024-03-25T18:37:20.841 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.35219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40746.log 2024-03-25T18:37:20.841 INFO:teuthology.orchestra.run.smithi155.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40629.log.gz 2024-03-25T18:37:20.841 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41605.log 2024-03-25T18:37:20.841 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40746.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.35219.log.gz 2024-03-25T18:37:20.841 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29355.log 2024-03-25T18:37:20.841 INFO:teuthology.orchestra.run.smithi155.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41761.log.gz 2024-03-25T18:37:20.841 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40238.log 2024-03-25T18:37:20.841 INFO:teuthology.orchestra.run.smithi155.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40746.log.gz 2024-03-25T18:37:20.842 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph.tmp-client.admin.29355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40355.log 2024-03-25T18:37:20.842 INFO:teuthology.orchestra.run.smithi155.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41605.log.gz 2024-03-25T18:37:20.842 INFO:teuthology.orchestra.run.smithi155.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29355.log.gz 2024-03-25T18:37:20.842 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41098.log 2024-03-25T18:37:20.842 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39223.log 2024-03-25T18:37:20.842 INFO:teuthology.orchestra.run.smithi155.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40238.log.gz 2024-03-25T18:37:20.842 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41098.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-25T18:37:20.842 INFO:teuthology.orchestra.run.smithi155.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40355.log.gz 2024-03-25T18:37:20.842 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40277.log 2024-03-25T18:37:20.842 INFO:teuthology.orchestra.run.smithi155.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41098.log.gz 2024-03-25T18:37:20.842 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39887.log 2024-03-25T18:37:20.842 INFO:teuthology.orchestra.run.smithi155.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39223.log.gz 2024-03-25T18:37:20.842 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35522.log 2024-03-25T18:37:20.842 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41215.log 2024-03-25T18:37:20.843 INFO:teuthology.orchestra.run.smithi155.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40277.log.gz 2024-03-25T18:37:20.843 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.35522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40043.log 2024-03-25T18:37:20.843 INFO:teuthology.orchestra.run.smithi155.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39887.log.gz 2024-03-25T18:37:20.843 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41722.log 2024-03-25T18:37:20.843 INFO:teuthology.orchestra.run.smithi155.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.35522.log.gz 2024-03-25T18:37:20.843 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40824.log 2024-03-25T18:37:20.843 INFO:teuthology.orchestra.run.smithi155.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41215.log.gz 2024-03-25T18:37:20.843 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41722.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40043.log.gz 2024-03-25T18:37:20.843 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41995.log 2024-03-25T18:37:20.843 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40824.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41722.log.gz 2024-03-25T18:37:20.843 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41488.log 2024-03-25T18:37:20.843 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36549.log 2024-03-25T18:37:20.843 INFO:teuthology.orchestra.run.smithi155.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40824.log.gz 2024-03-25T18:37:20.843 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42073.log 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41995.log.gz 2024-03-25T18:37:20.844 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-03-25T18:37:20.844 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.36549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38807.log 2024-03-25T18:37:20.844 INFO:teuthology.orchestra.run.smithi155.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41488.log.gz 2024-03-25T18:37:20.844 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.42073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41449.log 2024-03-25T18:37:20.844 INFO:teuthology.orchestra.run.smithi155.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.36549.log.gz 2024-03-25T18:37:20.844 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.38807.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40433.log 2024-03-25T18:37:20.844 INFO:teuthology.orchestra.run.smithi155.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.42073.log.gz 2024-03-25T18:37:20.844 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41449.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38807.log.gz 2024-03-25T18:37:20.844 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40082.log 2024-03-25T18:37:20.844 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-25T18:37:20.844 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40433.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41449.log.gz 2024-03-25T18:37:20.844 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39692.log 2024-03-25T18:37:20.844 INFO:teuthology.orchestra.run.smithi155.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40433.log.gz 2024-03-25T18:37:20.844 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37428.log 2024-03-25T18:37:20.844 INFO:teuthology.orchestra.run.smithi155.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40082.log.gz 2024-03-25T18:37:20.845 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40551.log 2024-03-25T18:37:20.845 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.37428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39340.log 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39692.log.gz 2024-03-25T18:37:20.845 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-03-25T18:37:20.845 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40551.log: gzip 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37428.log.gz 2024-03-25T18:37:20.845 INFO:teuthology.orchestra.run.smithi155.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41137.log 2024-03-25T18:37:20.845 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39340.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40551.log.gz 2024-03-25T18:37:20.845 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40668.log 2024-03-25T18:37:20.845 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41137.log: 86.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42112.log 2024-03-25T18:37:20.845 INFO:teuthology.orchestra.run.smithi155.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39340.log.gz 2024-03-25T18:37:20.845 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40668.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-25T18:37:20.845 INFO:teuthology.orchestra.run.smithi155.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41137.log.gz 2024-03-25T18:37:20.845 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41020.log 2024-03-25T18:37:20.845 INFO:teuthology.orchestra.run.smithi155.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40668.log.gz 2024-03-25T18:37:20.845 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.42112.log: /var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41566.log 2024-03-25T18:37:20.845 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39613.log 2024-03-25T18:37:20.846 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41566.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.42112.log.gz 2024-03-25T18:37:20.846 INFO:teuthology.orchestra.run.smithi155.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41020.log.gz 2024-03-25T18:37:20.846 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41566.log.gz 2024-03-25T18:37:20.846 INFO:teuthology.orchestra.run.smithi155.stderr: /var/log/ceph/ceph-client.admin.39770.log 2024-03-25T18:37:20.846 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39184.log 2024-03-25T18:37:20.846 INFO:teuthology.orchestra.run.smithi155.stderr: 86.0%/var/log/ceph/ceph-client.admin.39770.log: -- replaced with /var/log/ceph/ceph-client.admin.39613.log.gz 2024-03-25T18:37:20.846 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42229.log 2024-03-25T18:37:20.846 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39926.log 2024-03-25T18:37:20.846 INFO:teuthology.orchestra.run.smithi155.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39770.log.gz 2024-03-25T18:37:20.846 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39184.log: /var/log/ceph/ceph-client.admin.42229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42268.log 2024-03-25T18:37:20.846 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39926.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39184.log.gz 2024-03-25T18:37:20.846 INFO:teuthology.orchestra.run.smithi155.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.42229.log.gz 2024-03-25T18:37:20.846 INFO:teuthology.orchestra.run.smithi155.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39926.log.gz 2024-03-25T18:37:20.846 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.42268.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T18:37:20.847 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- 86.0% -- replaced with /var/log/ceph/ceph-client.admin.42268.log.gz 2024-03-25T18:37:20.847 INFO:teuthology.orchestra.run.smithi155.stderr: /var/log/ceph/ceph-client.admin.39418.log 2024-03-25T18:37:20.847 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42190.log 2024-03-25T18:37:20.847 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41059.log 2024-03-25T18:37:20.847 INFO:teuthology.orchestra.run.smithi155.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39418.log.gz 2024-03-25T18:37:20.847 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.42190.log: 92.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T18:37:20.847 INFO:teuthology.orchestra.run.smithi155.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.42190.log.gz 2024-03-25T18:37:20.847 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-25T18:37:20.847 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40121.log 2024-03-25T18:37:20.847 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41059.log.gz 2024-03-25T18:37:20.847 INFO:teuthology.orchestra.run.smithi155.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39104.log 2024-03-25T18:37:20.847 INFO:teuthology.orchestra.run.smithi155.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-25T18:37:20.847 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5/var/log/ceph/ceph-client.admin.40121.log: --verbose -- /var/log/ceph/ceph-client.admin.39379.log 2024-03-25T18:37:20.847 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42034.log 2024-03-25T18:37:20.847 INFO:teuthology.orchestra.run.smithi155.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40121.log.gz 2024-03-25T18:37:20.848 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39379.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39104.log.gz 2024-03-25T18:37:20.848 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39457.log 2024-03-25T18:37:20.848 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.42034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40942.log 2024-03-25T18:37:20.848 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39457.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39379.log.gz 2024-03-25T18:37:20.848 INFO:teuthology.orchestra.run.smithi155.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.42034.log.gz 2024-03-25T18:37:20.848 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41644.log 2024-03-25T18:37:20.848 INFO:teuthology.orchestra.run.smithi155.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39457.log.gz 2024-03-25T18:37:20.848 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40590.log 2024-03-25T18:37:20.848 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40942.log: /var/log/ceph/ceph-client.admin.41644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40903.log 2024-03-25T18:37:20.848 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40590.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40942.log.gz 2024-03-25T18:37:20.848 INFO:teuthology.orchestra.run.smithi155.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41644.log.gz 2024-03-25T18:37:20.848 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 86.1% --verbose -- /var/log/ceph/ceph-client.admin.41176.log 2024-03-25T18:37:20.848 INFO:teuthology.orchestra.run.smithi155.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40590.log.gz 2024-03-25T18:37:20.848 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40903.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-25T18:37:20.849 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41176.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40903.log.gz 2024-03-25T18:37:20.849 INFO:teuthology.orchestra.run.smithi155.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41176.log.gz 2024-03-25T18:37:20.894 INFO:teuthology.orchestra.run.smithi167.stderr: 92.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-25T18:37:21.629 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-mon.c.log: 93.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-25T18:37:21.659 INFO:teuthology.orchestra.run.smithi155.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-25T18:37:21.696 INFO:teuthology.orchestra.run.smithi167.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-25T18:37:21.699 INFO:teuthology.orchestra.run.smithi167.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-25T18:37:21.892 INFO:teuthology.orchestra.run.smithi155.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-25T18:37:21.916 INFO:teuthology.orchestra.run.smithi155.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-25T18:37:22.002 INFO:teuthology.orchestra.run.smithi167.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-25T18:37:22.041 INFO:teuthology.orchestra.run.smithi167.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-25T18:37:25.831 INFO:teuthology.orchestra.run.smithi167.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-25T18:37:25.832 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-03-25T18:37:25.832 INFO:teuthology.orchestra.run.smithi167.stderr:real 0m5.035s 2024-03-25T18:37:25.832 INFO:teuthology.orchestra.run.smithi167.stderr:user 0m8.982s 2024-03-25T18:37:25.833 INFO:teuthology.orchestra.run.smithi167.stderr:sys 0m0.325s 2024-03-25T18:37:25.938 INFO:teuthology.orchestra.run.smithi155.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-25T18:37:26.339 INFO:teuthology.orchestra.run.smithi155.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-25T18:37:26.340 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-03-25T18:37:26.341 INFO:teuthology.orchestra.run.smithi155.stderr:real 0m5.576s 2024-03-25T18:37:26.341 INFO:teuthology.orchestra.run.smithi155.stderr:user 0m14.160s 2024-03-25T18:37:26.341 INFO:teuthology.orchestra.run.smithi155.stderr:sys 0m0.545s 2024-03-25T18:37:26.341 INFO:tasks.ceph:Archiving logs... 2024-03-25T18:37:26.341 DEBUG:teuthology.misc:Transferring archived files from smithi155:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621273/remote/smithi155/log 2024-03-25T18:37:26.342 DEBUG:teuthology.orchestra.run.smithi155:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T18:37:28.811 DEBUG:teuthology.misc:Transferring archived files from smithi167:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621273/remote/smithi167/log 2024-03-25T18:37:28.812 DEBUG:teuthology.orchestra.run.smithi167:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T18:37:30.259 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:37:30.464 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=507f5d0c41934aa08accb1c19719558a Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:37:30.468 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-25T18:37:30.478 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-25T18:37:30.519 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-25T18:37:30.519 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T18:37:30.533 DEBUG:teuthology.orchestra.run.smithi167:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T18:37:30.615 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-25T18:37:30.623 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-25T18:37:30.624 DEBUG:teuthology.orchestra.run.smithi155:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:37:30.626 DEBUG:teuthology.orchestra.run.smithi167:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:37:30.640 INFO:teuthology.orchestra.run.smithi155.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:37:30.643 INFO:teuthology.orchestra.run.smithi167.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:37:30.644 INFO:teuthology.orchestra.run.smithi155.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:37:30.644 INFO:teuthology.orchestra.run.smithi155.stdout:=============================================================================== 2024-03-25T18:37:30.644 INFO:teuthology.orchestra.run.smithi155.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 25 +559us[ +558us] +/- 75ms 2024-03-25T18:37:30.644 INFO:teuthology.orchestra.run.smithi155.stdout:^* hv02.front.sepia.ceph.com 3 6 377 24 -1076us[-1078us] +/- 60ms 2024-03-25T18:37:30.645 INFO:teuthology.orchestra.run.smithi155.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 26 +795us[ +794us] +/- 65ms 2024-03-25T18:37:30.645 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:37:30.647 INFO:teuthology.orchestra.run.smithi167.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:37:30.647 INFO:teuthology.orchestra.run.smithi167.stdout:=============================================================================== 2024-03-25T18:37:30.647 INFO:teuthology.orchestra.run.smithi167.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 22 +530us[ +527us] +/- 75ms 2024-03-25T18:37:30.647 INFO:teuthology.orchestra.run.smithi167.stdout:^* hv02.front.sepia.ceph.com 3 6 377 17 -1086us[-1089us] +/- 60ms 2024-03-25T18:37:30.648 INFO:teuthology.orchestra.run.smithi167.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 32 +777us[ +773us] +/- 65ms 2024-03-25T18:37:30.648 INFO:teuthology.orchestra.run.smithi167.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:37:30.649 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-25T18:37:30.657 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-25T18:37:30.657 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-25T18:37:30.714 DEBUG:teuthology.orchestra.run.smithi155:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T18:37:30.787 DEBUG:teuthology.orchestra.run.smithi167:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T18:37:30.858 DEBUG:teuthology.orchestra.run.smithi155:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:37:30.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:37:30.890 DEBUG:teuthology.orchestra.run.smithi167:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:37:30.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:37:30.921 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-25T18:37:30.932 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-25T18:37:30.940 INFO:teuthology.task.internal:Duration was 7919.031526 seconds 2024-03-25T18:37:30.940 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-25T18:37:30.949 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-25T18:37:30.949 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T18:37:30.952 DEBUG:teuthology.orchestra.run.smithi167:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T18:37:30.994 INFO:teuthology.orchestra.run.smithi155.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:37:31.006 INFO:teuthology.orchestra.run.smithi167.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:37:31.380 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-25T18:37:31.380 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi155.front.sepia.ceph.com 2024-03-25T18:37:31.380 DEBUG:teuthology.orchestra.run.smithi155:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T18:37:31.399 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi167.front.sepia.ceph.com 2024-03-25T18:37:31.400 DEBUG:teuthology.orchestra.run.smithi167:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T18:37:31.422 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-25T18:37:31.423 DEBUG:teuthology.orchestra.run.smithi155:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T18:37:31.442 DEBUG:teuthology.orchestra.run.smithi167:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T18:37:31.491 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-25T18:37:31.491 DEBUG:teuthology.orchestra.run.smithi155:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T18:37:31.511 DEBUG:teuthology.orchestra.run.smithi167:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T18:37:31.635 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-25T18:37:31.644 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-25T18:37:31.645 DEBUG:teuthology.orchestra.run.smithi155:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T18:37:31.671 DEBUG:teuthology.orchestra.run.smithi167:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T18:37:31.698 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-25T18:37:31.708 DEBUG:teuthology.orchestra.run.smithi155:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:37:31.714 DEBUG:teuthology.orchestra.run.smithi167:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:37:31.737 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern = core 2024-03-25T18:37:31.764 INFO:teuthology.orchestra.run.smithi167.stdout:kernel.core_pattern = core 2024-03-25T18:37:31.791 DEBUG:teuthology.orchestra.run.smithi155:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:37:31.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:37:31.817 DEBUG:teuthology.orchestra.run.smithi167:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:37:31.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:37:31.846 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-25T18:37:31.855 INFO:teuthology.task.internal:Transferring archived files... 2024-03-25T18:37:31.856 DEBUG:teuthology.misc:Transferring archived files from smithi155:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621273/remote/smithi155 2024-03-25T18:37:31.856 DEBUG:teuthology.orchestra.run.smithi155:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T18:37:31.893 DEBUG:teuthology.misc:Transferring archived files from smithi167:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621273/remote/smithi167 2024-03-25T18:37:31.894 DEBUG:teuthology.orchestra.run.smithi167:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T18:37:31.929 INFO:teuthology.task.internal:Removing archive directory... 2024-03-25T18:37:31.930 DEBUG:teuthology.orchestra.run.smithi155:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T18:37:31.932 DEBUG:teuthology.orchestra.run.smithi167:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T18:37:31.978 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-25T18:37:31.989 INFO:teuthology.task.internal:Not uploading archives. 2024-03-25T18:37:31.989 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-25T18:37:31.998 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-25T18:37:31.998 DEBUG:teuthology.orchestra.run.smithi155:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T18:37:32.000 DEBUG:teuthology.orchestra.run.smithi167:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T18:37:32.012 INFO:teuthology.orchestra.run.smithi155.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 18:37 /home/ubuntu/cephtest 2024-03-25T18:37:32.033 INFO:teuthology.orchestra.run.smithi167.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 18:37 /home/ubuntu/cephtest 2024-03-25T18:37:32.034 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-25T18:37:32.043 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-25T18:37:32.062 INFO:teuthology.nuke:Checking targets against current locks 2024-03-25T18:37:32.084 DEBUG:teuthology.nuke:shortname: smithi155 2024-03-25T18:37:32.085 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T18:37:32.105 DEBUG:teuthology.nuke:shortname: smithi167 2024-03-25T18:37:32.105 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T18:37:32.109 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi155.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621273', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:16:04.241597', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG9GnDhi3qlVC9dImWU7K2yGAWDRaZmta/anqc1QhHI2yQmtpqA6J5ARO+7/Ii0+YsV6tYMdaqBJUklrT1zU160='} 2024-03-25T18:37:32.125 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi167.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621273', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:16:04.243555', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGkR3F73DIi0cpF43Ju4TRu8aT0j6LuPuo7GMORA5vpzJkhzAQzfFd85n98iFpyhxZ9s35yGY/5pPQTjWg7QfRc='} 2024-03-25T18:37:32.154 INFO:teuthology.orchestra.console.smithi155:Power off 2024-03-25T18:37:32.154 DEBUG:teuthology.orchestra.console.smithi155:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T18:37:32.177 DEBUG:teuthology.orchestra.console.smithi155:power off output: Chassis Power Control: Down/Off 2024-03-25T18:37:32.178 DEBUG:teuthology.orchestra.console.smithi155:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:37:32.193 INFO:teuthology.orchestra.console.smithi167:Power off 2024-03-25T18:37:32.193 DEBUG:teuthology.orchestra.console.smithi167:pexpect command: ipmitool -H smithi167.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T18:37:32.203 DEBUG:teuthology.orchestra.console.smithi155:check power output: Chassis Power is on 2024-03-25T18:37:32.216 DEBUG:teuthology.orchestra.console.smithi167:power off output: Chassis Power Control: Down/Off 2024-03-25T18:37:32.216 DEBUG:teuthology.orchestra.console.smithi167:pexpect command: ipmitool -H smithi167.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:37:32.237 DEBUG:teuthology.orchestra.console.smithi167:check power output: Chassis Power is on 2024-03-25T18:37:36.204 DEBUG:teuthology.orchestra.console.smithi155:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:37:36.238 DEBUG:teuthology.orchestra.console.smithi167:pexpect command: ipmitool -H smithi167.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:37:36.314 DEBUG:teuthology.orchestra.console.smithi155:check power output: Chassis Power is on 2024-03-25T18:37:36.347 DEBUG:teuthology.orchestra.console.smithi167:check power output: Chassis Power is on 2024-03-25T18:37:40.318 DEBUG:teuthology.orchestra.console.smithi155:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:37:40.348 DEBUG:teuthology.orchestra.console.smithi167:pexpect command: ipmitool -H smithi167.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:37:40.430 DEBUG:teuthology.orchestra.console.smithi155:check power output: Chassis Power is on 2024-03-25T18:37:40.458 DEBUG:teuthology.orchestra.console.smithi167:check power output: Chassis Power is off 2024-03-25T18:37:40.559 INFO:teuthology.orchestra.console.smithi167:Power off completed 2024-03-25T18:37:44.433 DEBUG:teuthology.orchestra.console.smithi155:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:37:44.544 DEBUG:teuthology.orchestra.console.smithi155:check power output: Chassis Power is off 2024-03-25T18:37:44.644 INFO:teuthology.orchestra.console.smithi155:Power off completed 2024-03-25T18:37:44.749 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} duration: 7919.031525850296 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=507f5d0c41934aa08accb1c19719558a status: fail success: false 2024-03-25T18:37:44.749 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T18:37:44.828 INFO:teuthology.run:FAIL